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A  REVISED  MODIFIED  PARALLEL  ANALYSIS  (RMPA)  FOR  THE 
CONSTRUCTION  OF  UNIDIMENSIONAL  ITEM  POOLS 


BACKGROUND 

The  increasing  popularity  of  Item  Response  Theory  (IRT)  (e.g.  Hambleton,  1983;  Hulin, 
Drasgow  &  Parsons  1983;  Lord,  1980)  in  educational,  personnel  and  psychological  testing 
has  caused  a  revolution  in  this  domain.  These  new  models  enable  researchers  and  test  users  to 
solve  efficiently  otherwise  intractable  problems  and  develop  many  innovative  testing 
procedures. 

Perhaps  the  most  promising,  and  undoubtedly  the  most  intriguing,  one  is  Computerized 
Adaptive  Testing  (CAT).  The  basic  ideas  as  well  as  the  theoretical  and  practical  advantages  of 
CAT  are  well  known  and  widely  acknowledged  (e.g.  Green,  1983;  Weiss,  1983).  The 
increasing  availability  and  acceptance  of  computers  in  everyday  life  and  their  lower  prices 
make  CAT  a  feasible  alternative  to  traditional  forms  of  testing. 

Why  is  it,  then,  that  CAT  is  relatively  slow  in  replacing  conventional  testing  procedures  ?  One 
possible  reason  are  the  various  problems  related  to  the  construction,  validation  and 
maintenance  of  the  large  item  pools  required  by  this  new  testing  protocol. 

From  a  psychometric  point  of  view  one  of  the  most  interesting  and  challenging  problems  is  the 
assessment  of  the  pools'  dimensionality.  Though  multidimensional  item  response  models 
have  been  developed  (e.g.  Reckase,  1985;  Sympson,  1978),  most  readily  applicable  IRT 
nnodels  used  today  assume  that  the  test  takers  responses  to  all  items  depend  on  a  single  latent 
Uait  (ability).  Thus,  it  is  crucial  to  establish  that  any  item  used  in  estimating  the  examinee's 
position  along  this  ability  continuum  measures,  in  fact,  the  same  trait.  In  other  words,  the 
need  to  demonstrate  that  a  given  item  pool  is  truly  unidimensional  is  a  necessary  condition  for 
its  use  in  CAT. 

Defining  and  Assessing  Unidimensionality 

Consider  a  test  consisting  of  n  items  selected  from  a  larger  item  pool.  Let  Uj  be  the  vector  of  n 
binary  responses  to  the  test's  items  (taking  values  of  1  and  0  for  correct  and  incorrect 
response,  respectively),  generated  by  the  ith  test  taker  (i=l...N),  and  let  U..  be  her  response 
to  the  jth  item  (j=  1  ...n).  Finally,  let  O;  be  a  vector  of  t  latent  traits  characterizing  the 
examinee's  ^ilities.  The  strong  principle  of  local  independence  (McDonald,  I9&  i )  states  that; 


This  principle  asserts  that  the  responses  to  any  pair  of  items  are  statistically  mutually 
independent  for  any  individual,  or  any  subpopulation  with  fixed  latent  traits.  The 
dimensionality  of  U  is,  simply,  the  minimal  number  of  latent  traits  necessary  to  produce  a 
(strong)  locally  independent  model  for  U.  Thus,  a  pool  is  unidimensional  if  responses  to  all  its 
items  can  be  produced  by  unidimensional  locally  independent  models. 

Although  a  voluminous  literature  exists  on  the  issue  of  unidimensionality  of  items  and  tests 
(see  Berger  and  Knol,  1990;  Hattie,  1984  and  1985  for  partial  reviews),  current'y  there  is  no 
single  approach  which  is  fully  satisfactory  and/or  universally  accepted.  Hattie  (1984) 
compiled  a  list  of  87  measures  of  unidimensionality  and  classified  them  into  five 
nonoverlapping  classes  according  to  their  underlying  rationale.  He  distinguished  between 
indices  based  on 

(i)  closeness  to  specific  answer  patterns, 

(ii)  reliability  coefficients, 

(iii)  principal  components  (PC), 

(iv)  factor  analysis  (FA)  and 

(v)  goodness  of  fit  to  various  IRT  models. 

Hattie  questioned  the  theoretical  rationale  of  indices  based  on  response  patterns  and  reliability 
and  showed  empirically  that  the  measures  based  on  PC,  FA  and  one  parameter  IRT  (the  Rasch 
model)  are  outperformed  by  methods  quantifying  deviation  from  multi-parameter  IRT  models. 


"Approximate  "  Unidimensionality 

Many  researchers  have  argued,  based  on  theoretical  and  empirical  observations,  that  purely 
unidimensional  tests,  or  pools,  are  quite  rare  ( e.g.  Ackerman,  1989;  Humphreys,  1985; 
Reckase,  Ackerman  &  Carlson,  1988;  Traub,  1983;  Yen,  1984,  1985).  If,  in  fact, 
unidimensionality  is  frequently  violated  it  is  important  to  determine  the  practical  implications  of 
such  violations.  Following  Reckase’s  original  work  (1979),  several  researchers  (e.g. 

Drasgow  &  Parson,  1983;  Yen,  1984,  1985)  have  shown  that  unidimensional  models  are 
quite  robust  under  multidimensionality  as  long  as 

(i)  There  is  a  single  "dcrr.inant"  factor,  and 

(ii)  Item  difficulty  is  not  confounded  with  dimensionality. 
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These,  and  other  similar,  studies  suggest  that  strict  unidimensionai  pools  are  not  necessary  for 
many  practical  applications  of  unidimensional  IRT  models  (e.g.  CAT).  It  is,  however, 
impoitant  to  develop  methods  that  can  identify  pools  which  are  almost  /  practically  / 
approximately  unidimensional  (i.e.  they  deviate  from  strict  unidimensionality  to  a  degree 
which  does  not  seriously  affect  the  fit  or  accuracy  of  the  unidimensionai  IRT  model). 

This  is  the  motivation  behind  recent  work  by  Stout,  who  developed  a  test  of  the  essential 
unidimensionality  of  a  d&ta  set  (Stoat,  1987,1990;  Nandakumar,  1991).  Essential 
independence  is  achieved  if  the  mean  covariance  (conditional  on  Bj,  the  test  taker's  vector  of  t 
latent  traits)  between  all  n(n-l)/2  pairs  of  items  approaches  0  as  the  number  of  items  increases 
to  infinity,  and  the  essential  dimensionality  of  a  pool  is  the  smallest  number  of  latent  traits 
necessary  to  satisfy  essential  independence.  Essential  inctependence  is  a  weaker  requirement 
than  strong  local  independence  and,  in  practice,  it  is  obtained  whenever  there  is  a  single 
dominant  dimension  in  the  data  (e.g.  Nandakumar,  1991). 

In  the  same  spirit  Drasgow  and  Lissak(1983)  presented  Modified  Parallel  Analysis  (MPA  for 
short)  as  "a  technique  that  can  determine  when  an  item  pool  is  sufficiently  unidimensionai  for 
the  use  of  IRT"  (Drasgow  and  Lissak,  1983,  page  36S).  Modified  Parallel  Analysis  relies  on 
FA,  a  well  understood  method  which  is  widely  available  to  users  in  most  statistical  packages. 
Thus,  it  is  (conceptually  and  computationally)  easier  to  use  than  Stout's  methods.  This  study 
will  develop  a  revised  and  improved  version  of  MPA. 

Parallel  and  Modified  Parallel  Analysis 

Parallel  Analysis  (PA)  was  proposed  by  Horn  ( 196S)  as  an  alternative  to  traditional  factor 
analytical  methods  for  identifying  the  number  of  latent  factors.  The  standard  methods  are 
based  on  various  functions  of  the  eigenvalues  of  the  correlation  matrix.  Among  them,  the 
eigenvalues’  absolute  size  (e.g.  Kaiser,  1960),  their  overall  pattern  (e.g.  Cattell,  1966),  or 
tlwir  distribution  under  the  multivariate  normal  model  (e.g.  Bartlett,  1950). 

The  rationale  behind  PA  is  intuitively  compelling,  and  its  application  is  simple  and 
straightforward:  Random  correlation  matrices  are  generated,  and  their  eigenvalues  are 
extracted  and  averaged.  The  eigenvalues  of  the  actual  correlations  are  compared  to  these 
means  and  those  factors  with  eigenvalues  larger  than  their  counterparts  from  the 
randomly  generated  data  are  retained.  Crawford  and  Koopman  (1973),  Humphreys  and 
Montanelli  (1975)  and  Zwick  and  Vclicer  (1986),  among  others,  report  that  PA  works  well  in 
both  Principal  Components  (PC)  and  Factor  Analysis  (FA),  Recently  Longman,  Cota,  Holden 
and  Fekken  (1989)  published  regression  equations  that  eliminate  the  need  to  actually  generate 
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random  matrices  for  each  PA  (for  the  PC  case). 

ParaUel  Analysis  is  used  to  determine  the  true  dimensionality  of  a  given  data  set,  whereas  in 
most  applications  of  CAT  one  seeks  to  determine  whether  a  data  set  deviates  significantly  from  ^ 
unidimensionality.  Modified  Parallel  Analysis  (Drasgow  &  Lissak,  1983)  provides  an 
ingenious  way  of  answering  this  question,  using  the  rationale  of  PA.  Its  basic  stages  are: 

f 

( 1)  The  intercorrelations  (preferably  tetrachoric)  of  the  test's  items  are  factor  analyzed  and  the 
eigenvalues  of  the  unrotated  solution  are  calculated. 

(2)  A  "parallel"  unidimensional  data  set  is  generated  by  an  IRT  model.  This  data  set  parallels 
the  observed  one  along  all  its  attributes:  It  has  an  equal  number  of  examinees  with  identical 
abilities,  and  it  has  the  same  number  of  items  with  identical  parameters.  Since  responses 
are  generated  by  an  unidimensional  IRT  model  satisfying  the  strong  local  independence 
principle  the  data  set  is,  by  definition,  unidimensional. 

(3)  The  (tetrachoric)  correlations  of  the  parallel  data  set  are  factor  analyzed,  and  the 
eigenvalues  of  the  unrotated  solution  are  calculated. 

(4)  The  dimensionality  of  the  pool  is  assessed  by  comparing  the  magnitude  of  the  second 

♦ 

eigenvalues  of  the  two  data  sets:  If  the  actual  value  (calculated  in  stage  I)  is  "sufficiently 
close"  to  the  one  obtained  from  the  parallel  data  set  (calculated  in  stage  3),  the  test  is 
unidimensional.  ' 

Drasgow  and  Lissak  (1983)  recommend  that  the  items'  commonalities  be  estimated  by  the 
largest  (absolute)  off-diagonal  correlation,  and  suggest  an  ad  hoc  procedure  for  imputation  of 
tetrachoric  correlations  for  those  cases  where  the  regular  algorithm  fails  to  converge.  They 
also  report  five  empirical  studies  providing  strong  empirical  support  for  the  procedure. 

Eigenvalue  based  factor  analytical  techniques  are  not  always  successful  in  recovering  the  true 
dimensionality  of  binary  data  and,  consequently,  can’t  always  distinguish  between 
unidimensional  and  multidimensional  data  sets  (e.g.  Collins,  Cliff,  McCormick  and  2^tlin, 

1986;  Hattie,  1984;  Knol  and  Berger,  1991;  Roznowsky,  Tucker  &  Humphreys,  1991; 

Zwick  and  Velicer,  1986).  Thus  it  may  seem  surprising  that  some  of  the  same  measures 

t 

perform  very  well  in  the  framework  of  PA,  and  MPA.  It  is  important  to  stress  that  the  key  to 
the  success  of  these  methods  is  their  comparative  nature.  Whatever  deficiencies  these  statistics 
have,  they  affect  equally  the  results  of  the  two  data  sets.  Both  PA  and  MPA  focus  on,  and  ^ 
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highJight,  whatever  differences  exist  between  the  empirical  and  parallel  data  sets  above  and 
beyond  the  systematic  biases  that  the  FA  based  measures  may  share. 

Thus,  in  Hattie's  (1984)  typology  MPA  should  not  be  considered  a  "factor  analytic  approach". 
In  fact,  it  is  closer  to  the  "measures  of  fit  to  IRT  models".  MPA  is  a  general  method  for 
assessing  the  similarity,  or  closeness,  between  two  parallel  data  sets  (one  of  which  is  known 
to  be  unidimensional),  in  which  the  similarity  is  quantified  by  some  of  the  statistics  usually 
employed  in  FA. 

A  critique  of  MPA 

Modified  Parallel  Analysis  suffers  from  a  few  technical  limitations.  In  this  section  we 
describe  these  limitations  and  the  problems  they  may  cause  in  applying  the  method: 

(i)  MPA  is  a  randomized  procedure,  i.e.  its  results  depend  to  a  certain  degree  on  a  random 
process  which  is  totally  unrelated  to  the  process  of  interest,  namely,  the  selection  of  the 
parallel  data  set.  Thus,  with  small  enough  samples,  researchers  applying  exactly  the  same 
procedure  to  the  same  set  of  data  may  reach  different  conclusions  because  of  the  variance 
between  the  random  data  sets  generated  in  their  simulations. 

(ii)  The  simulated  and  the  empirical  data  sets  are  equated  along  most  important  dimensions  and 
any  discrepancy  between  their  eigenvalues  can  ,supposedly,  be  attributed  to  the 
multidimensionality  of  the  empirical  matrix.  Yet,  the  communalities  are  estimated  in  a 
purely  empirical  fashion  separately  for  each  data  set,  introducing  another  important 
difference  between  them.  This  factor  may  bias  (in  an  unknown  direction  and  to  an 
unknown  degree)  the  comparative  analysis. 

(iii)  MPA  is  a  heuristic  procedure,  i.e.  it  lacks  a  measure  of  sampling  variability  for  the  formal 
assessment  of  the  closeness  of  the  critical  statistic  (the  second  eigenvalue)  obtained  from 
the  unidimensional  and  the  empirical  solutions. 

Other  important  limitations  of  MPA  are; 

(iv)  It  compares  only  the  second  pair  of  eigenvalues  of  the  two  matrices.  This  choice  lacks  a 
solid  theoretical  or  empirical  justification,  and  it  may  miss  differences  between  the  other 
eigenvalues  (especially  the  third). 
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(v)  MPA  is  too  limited  in  its  scope.  The  technique  provides  a  global  omnibus  lest  of  the 
hypothesis  concerning  the  pool’s  unidimensionality.  It  lacks,  however,  a  mechanism  to 
follow  up  rejections  of  the  hypothesized  pattern,  by  eliminating  some  items  and  identify  a 
unidimensional  subset  of  the  pool. 

A  REVISED  MODIFIED  PARALLEL  ANALYSIS  (RMPA) 

In  this  section  we  outline  a  revised  procedure  (RMPA)  which  extends  and  generalizes  the 
MPA.  The  revised  method  offers  solutions  to  the  technical  problems  described  above 
and  incorporates  them  into  the  existing  framework  of  MPA.  Originally,  MPA  was  developed 
as  a  global  procedure  that  distinguishes  between  (e.ssentially)  unidimensional  tests  and 
multidimensional  ones.  RMPA  complements  this  aspect  by  a  second  stage  which  allows  one 
to  extract  unidimensional  subsets  from  larger,  potentially  multidimensional,  pools. 

To  solve  the  first  problem  we  replace  the  random  generation  of  a  parallel  unidimensional 
population  by  the  theoretical  derivation  of  the  expected  correlations  under  the  assumptions  of 
(1)  local  independence,  (2)  unidimensionality  of  the  parameter  space  and  (3)  the  three 
parameter  logistic  model  (e.g.  Lord,  1980).  The  probability  of  a  correct  response  for  item)  by 
a  test  taker  with  (a  single)  ability  0i  is  given  by  P(Uij  =  Il0i)  or,  in  a  shorter  notation,  Pjj; 

p..  =  c-  + _ '  ~  _ 

^  I  +  exp{-l.7aj(0j- bj)) 

where  a^  is  the  item's  discrimination  parameter,  b^  is  the  item’s  difficulty  and  c^  is  its  pseudo¬ 
guessing  probability  (see  Hambleton,  1983  or  Lord,  1980  for  details). 

Under  these  assumptions  the  expected  number  of  correct  answers  to  any  pair  of  arbitrary 
items,  J  and  k,  in  a  random  sample  of  N  examinees  is: 


(2) 


(4) 
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Given  f  and  the  two  marginals,  f.  and  the  expected  2x2  contingency  table  can  be 
constructed,  and  the  expected  tetrachoric  correlation  can  be  estimated  by  standard  methods 
i  (e.g.  by  solving  a  polynomial  using  the  Newton  Raphson  method,  as  suggested  by  Kendall  & 
Stuart  1979,  pages  324-327).  All  expectations  are  (as  in  the  original  MPA)  conditional  upon 
the  abilities  and  item  parameters  shared  by  the  two  data  sets.  The  calculation  can  be  further 
refined  when  the  true  distribution  of  the  unidimensional  abilities  (6i)  in  the  population  is 
known.  In  these  cases,  the  summation  is  replaced  by  integration  across  all  values  of  Bj 
weighted  according  to  the  probability  density  of  the  6j,  to  yield  the  matrix  of  expected 
tetrechoric  correlations  in  the  population. 

To  solve  the  second  problem  we  replace  the  separate  estimation  of  the  communalities  in  the 
two  data  sets  by  the  expected  tetrachoric  correlation  between  (hypothetical)  experimentally 
independent  administrations  of  any  item  under  the  assumptions  of  ( 1 )  local  independence, 

(2)  unidimensional  ability  and  (3)  a  three  parameter  logistic  item  curve.  This  procedure 
amounts  to  estimating  the  items'  communalities  by  their  expected  test-retest  reliabilities.  It  is 
well  known  (e.g.  Lord  &  Novick,  1968;  Mulaik,  1972)  that  a  measure's  reliability  provides  an 
upper  bound  to  its  communality.  The  estimation  procedure  is  just  a  special  case  of  the 
technique  described  above  for  the  calculation  of  the  expected  correlation.  More  specifically,  if 
we  let  jsk,  Equation  4  is  reduced  to: 


fji*  X  ■  (4a) 

i  =  1 

The  solution  of  the  third  problem  relies  on  a  data  analytic  procedure  known  as  "jacknifmg" 
(see  Arvesen  and  Salsburg,  1975,  Miller,  1974  or  Mosteller  &  Tukey,  1977  for  partial 
reviews)  ’.  Assume  that  the  original  nxn  correlation  matrix  between  the  test's  items  is  strictly 
unidimensional.  By  eliminating  one  item  at  a  time  (i.e.  deleting  a  row,  and  the  corresponding 
column,  from  the  original  matrix)  we  obtain  n  submatrices  of  order  (n- 1  )x(n- 1 ) 
which,  by  definition,  are  also  unidimensional.  Furthermore,  it  is  easy  to  show  that  under  the 
"one  factor  model"  (i.e.  a  matrix  of  rank  one),  the  average  first  eigenvalue  of  these  n 
submatrices,  scaled  by  a  factor  of  n/(n- 1),  is  an  unbiased  estimate  of  the  first  eigenvalue  of  the 
original  intact  matrix. 

An  useful  and  important  consequence  of  the  "eliminate  one  item  at  a  time"  procedure  is  that  it 
provides  a  simple  method  for  assessing  the  impact,  or  influence^,  of  any  single  item  on  the 
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test's  eigenvalues.  The  logic  of  the  MPA  procedure  predicts  that,  under  linidimensionalily,  the 
two  matrices  will  have  equal  eigenvalues.  For  example,  it  is  generally  accepted,  and  it  was 
confirmed  empirically  by  Drasgow  &  Lissak(  1983),  that  the  first  eigenvalue  (X.) )  is 
approximately  equal  in  the  observed  and  the  expected  matrices,  regardless  of  the 
dimensionality  of  the  observed  responses.  Thus,  except  for  sampling  error,  the  ratio  of  the 
two  eigenvalues,  RLp  should  be; 

RL,=  Xi(observed)/ X.i(expectcd)  =  1  .  (5) 

Furthermore,  under  unidimensionality,  the  eigenvalues  of  the  n  submatrices  of  the  two  data 
sets  will  be  similar,  will  have  equal  variances  and  will  be  highly  correlated.  Finally,  the 
removal  of  any  given  item  from  the  pool  will  affect  the  observed  and  the  expected  data  sets  in 
identical  fashion  and  to  an  equal  degree.  Thus,  equality  (5)  should  also  hold  in  all  n 
submatrices  obtained  by  eliminating  one  item  at  a  time.  Let  X.','  be  the  first  eigenvalue  of  the 
submatrix  obtained  after  the  deletion  of  item  i,  and  let  RL','  be  the  ratio  of  the  eigenvalues  from 
the  two  parallel  data  sets.  Then,  for  all  items  (i=!...n),  the  ratio  of  the  jacknifed 
eigenvalues  should  equal  the  ratio  of  the  original  values: 

RL’,'  =  X,’,'  (observed)/X.', '(expected)  =  RL,  .  (6) 

If  the  responses  are  unidimensional,  similar  results  are  expected  to  hold  for  the  second,  third, 
and  all  subsequent  eigenvalues.  If,  on  the  other  hand,  the  observed  responses  violate 
unidimensionality,  the  analysis  of  the  two  data  sets  should  yield  differential  results.  For 
example,  Drasgow  and  Liss£dt(l983)  based  the  original  MPA  on  the  prediction  that  the  second 
eigenvalue  of  the  observed  matrix  will  be  larger  than  its  counterpart  from  the  parallel 
unidimensional  data  set: 

RL2=  X-jCobserved)/  X.2(expecled)  >  1  .  (7) 

If  the  data  are  generated  by  a  multidimensional  model  we  expect  the  mean  of  the  n  second 
eigenvalues  extracted  from  the  observed  submatrices  to  be  larger,  and  their  variance  to 
be  higher,  than  their  counterparts  from  the  expected  data  .set.  Depending  on  the  type  and 
degree  of  deviation  from  unidimensionality,  the  correlation  between  the  observed  and 
expected  values  can  be  low  (or  even  negative).  Furthermore,  the  eigenvalues  of  the  observed 
respon.ses  will  be  more  .sensitive  to  the  removal  of  the  foreign  (or  "contaminating")  items. 
Since  the  expected  matrix  is  unidimcnsional,  its  eigenvalues  should  not  be  affected 
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considerably  when  any  arbitrary  item  is  removed.  However,  when  a  contaminating 
item  is  removed  from  a  multidimensional  test,  the  data  set  becomes  closer  to  unidimensionality 
*  and  its  eigenvalues  should  decrease.  For  example,  in  a  test  of  length  n=50  with  8  foreign 
items  (8/50=16%  contamination),  after  the  removal  of  such  an  item,  the  level  of  contamination 
is  reduced  to  (7/49=)  14%.  Thus,  whenever  a  contaminating  item  is  eliminated  the  matching 
eigenvalues  should  be  more  similar  to  each  other  than  in  those  cases  in  which  a  regular 
(noncontaminating)  item  is  removed.  Consequently,  the  ratio  of  the  eigenvalues  should  be 
closer  to  unity  in  these  instances. 

To  summarize,  for  any  given  data  set,  the  ratio  between  the  first  eigenvalues,  RLj,  in  the  two 
data  sets  can  be  used  as  a  benchmark  against  which  one  can  assess  and  lest  the  ratios  derived 
from  the  second  and  third  eigenvalues  (RL^  and  RL^,  respectively).  At  the  global  (i.e.  test  or 
pool)  level,  this  approach  is  attractive  because  the  behavior  of  RL2  and  RL^  is  assessed  by  a 
data  based  index  which  is  more  sensitive  to,  and  reflects,  the  peculiarities  and  idiosyncrasies  of 
the  specific  test  being  examined.  At  the  local  (i.e.  item)  level,  this  procedure  provides  a  natural 
way  of  ranking,  and  scaling,  the  items  according  to  their  deviation  from  the  pattern  expected 
under  unidimensionality.  These  properties  can  be  used  to  develop  a  procedure  for  testing  the 
global  dimensionality  of  the  observed  responses,  and  a  method  of  selecting  unidimensioaal 
^  pools.  In  the  next  section  we  describe  the  technical  details  of  such  a  testing  procedure. 

The  "gap  test" 

As  described  above,  we  propose  to  jacknife  the  two  parallel  correlation  matrices  and  calculate 
the  eigenvalues  of  all  n  submatrices.  To  facilitate  the  comparison  of  the  two  data  sets  we 
calculate,  for  all  items  (i=l...n)  and  for  the  first  k  eigenvalues  (typically  k=l,2,3  should 
suffice),  the  ratio  of  the  two  matched  eigenvalues: 

RLjj'  =  Xjj'(observed)/  X^'(expected)  (8) 

The  global  ratio  RL|,  as  well  as  the  individual  RL/  (i=l..n),  are  insensitive  to  the 
dimensionality  of  the  observed  data  s'*!.  Their  empirical  distribution  will  be  used  to  test  the 
hypothesis  that  the  ratios  of  the  second  and  third  eigenvalues  behave  similarly.  Formally,  we 
wish  to  test  that  F { RLj' )  =  F{  RL',' ) ,  and  F ( RLj' )  =  F { RLj' ) ,  where  F(  • )  stands  for  the 
^  distribution  of  the  relevant  statistic.  The  alternative  hypothesis  is  that  the  ratios  are  distributed 
differentially. 

^  We  are  particularly  interested  in  the  case  where  an  essentially  unidimensional  data  set  is 
contaminated  by  a  second  (sometimes  called  "nuisance")  ability.  We  speculated  earlier,  that 
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removal  of  such  contaminating  items  will  affect  differentially  the  two  matched  eigenvalues 
When  analyzing  the  correlations  from  the  observed  responses  we  expert  tr>  observe  two 
distinct  clusters  of  eigenvalues  —  from  the  unidimensional  and  the  contaminating  pool, 
respectively  —  separated  by  a  substantial  "jump".  No  parallel  clustering  and  separation  is 
expected  in  the  corresponding  eigenvalues  of  the  matrix  of  expected  correlations. 

To  detect  such  unusual  jumps  we  adopt  a  procedure  described  by  Wainer  and  Schacht  (1978) 
under  the  name  of  "gapping"  since  its  goal  is  to  detect  unusually  large  gaps  in  strings  of 
ordered  values.  The  first  step  in  this  procedure  is  to  rank  order  the  values  in  descending  order 
and  to  calculate  the  (n- 1 )  gaps,  g^.  by  subtracting  each  observation  from  the  immediately 
previous  (i.e.  larger)  one.  The  gaps  are  then  weighted  by  a  set  of  logistic  weights  to  yield 
weighted  gaps,  y..  These  weights  were  selected  to  account  and  compensate  for  the  fact  that, 
typically,  observations  are  mote  dense  (hence  should  be  overweighted)  near  the  center  and 
more  sparse  (and  should  be  underweighted)  in  the  tails  of  the  distribution.  Formally; 

Yi  =  Vi  (n  -  i)  gi  .  (9) 

Finally,  these  values  are  standardized  by  division  by  y^,  the  midmean  (i.e.  the  mean  of  the 
central  50%  values)  of  the  weighted  gaps.  Thus,  the  standardized  weighted  gaps  (SWGs 
for  short),  z.  can  be  expressed  as: 


Zi  =  y/y^.  (10) 

Zero  gaps  indicate  that  two  adjacent  observations  are  equal,  and  unit  gaps  indicate  that  the 
distance  between  two  observations  is  equal  to  the  gaps'  midmean.  By  definition,  all  gaps  are 
non-negative  but  are  unbounded  from  above.  Wainer  and  Schacht  (1978)  suggest  that  z. 
values  greater  than  2.25  indicate  "unusually"  large  gaps.  The  probability  of  observing  gaps 
this  wide  by  chance  is  approximately  0.03  under  the  normal  distribution,  but  this  value  was 
shown  by  Wainer  and  Schacht  (1978)  to  work  quite  well  for  a  variety  of  symmetric  t 
distributions  with  tails  larger  than  the  normal. 

We  will  use  this  procedure  to  detect  the  location  of  the  gap  separating  the  items  from  the  two 
pools,  on  the  basis  of  ratios  of  the  matched  eigenvalues,  RLjj'  (k  >  1).  Thus,  the  hypothesis 
will  be  tested  by  comparing  MAXfZj^j),  the  largest  SWG,  with  a  critical  rejection  threshold. 
However,  in  the  absence  of  precise  information  regarding  the  form  of  the  distribution  of  these 
ratios,  and  the  multiplicity  of  tests  involved,  it  is  not  sufficient  to  rely  on  the  2.25  universal 
rule  of  thumb  proposed  by  Wainer  and  Schacht.  Instead,  we  find  it  necessary  to  develop  more 


general  {and  more  conservative^)  rejection  mles  . 


There  arc  various  ways  of  deriving  critical  reject  on  points  for  this  decision:  If  the  distribution 
of  RL'i*  is  known  (e.g.  normal),  the  critical  values  can  be  obtained  from  the  appropriate  table. 
Otherwise,  one  can  estimate  the  desired  percentiles  (.01,  .05,  etc.)  from  the  distribution  of 
RLj'.  Finally,  one  can  use  a  version  of  Chebyshev  inequality  (e  g.  Stuart  and  Ord,  1987,  page 
1 10).  The  regular  Chebyshev  inequality  states  that  the  probability  of  finding  a  value  located 
more  than  K  standard  deviations  (SDs)  from  the  population's  mean  is  smaller  than  1/K^,  for 
any  distribution  with  finite  moments;  A  tighter  version,  invoking  the  additional  assumptions 
that  the  distribution  is  symmetric  and  unimodal,  yields  a  lower  upper  limit  (4/9K^),  for  the 
probability  of  the  same  event 

The  decision,  to  reject  H^,  will  be  based  on  a  comparison  with  a  critical  tlireshold,  T(2j).  The 
threshold  is  derived  from  the  distribution  of  the  ratios  of  the  first  eigenvalue,  RL'|',  in  the  same 
data  set.  Specifically,  for  k=2,3  we  will  reject  if: 

MAX(Z^j)>T(z,)  =  (M,  +  KS,) 

where  Mj  and  Sj  arc  the  mean  and  SD,  respectively,  of  the  SWGs,  z^,  calculated  from  the 
ratios  of  first  set  of  matched  eigenvalues,  RL'|'.  For  the  three  possible  distributional 
assumptions  described  above,  and  with  probability  of  Type  I  errors  fixed  at  0.01, 0.05  and 
0. 10,  K  takes  the  values  described  in  the  following  table: 


Prob  (Type  I  error) 

Assumption 

0.01 

0.05 

0.10 

Normality 

2.50 

2.00 

1.65 

Symmetry  +  unimodality 

6.67 

3.00 

2.11 

None 

10.00 

4.50 

3.  i  - 

The  normal  case  is  fully  consistent  with  Wainer  and  Schacht's  2.25  universal  rule  of  thumb, 
and  needs  no  further  elaboration.  It  is  included  in  the  table,  primarily,  as  a  benchmark  against 
which  the  more  conservative  Chebyshev  rules  can  be  evaluated.  We  will  have  more  to  say 
about  the  various  rejection  rules  later  in  the  paper. 
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A  ten  •  step  summary  of  RMPA 

( 1 )  Fo!  ,ving  the  administration  of  a  test  consisting  of  n  items  to  a  sample  of  N  test  takers, 
estimate 

(i)  the  three  parameters  of  each  item, 

(ii)  the  ability  of  each  examinee,  and 

(iii)  the  nxn  matrix  of  tetrachoric  correlations  between  the  test's  items. 

(2)  Using  the  ability  and  item  parameters  estimated  from  the  observed  responses,  calculate  the 
nxn  matrix  of  expected  tetrachoric  correlations  between  the  items. 

(3)  The  (unit)  diagonal  values  of  the  observed  and  expected  correlation  matrices  are  replaced 
by  the  expected  item  test-retest  reliabilities,  and  the  first  k  (k=  1,2,3)  eigenvalues  of  the  two 
matrices  are  extracted. 

Except  for  a  few  technical  refinements  the  previous  steps  are  identical  to,  and  allow  the 
application  of,  MPA. 

(4)  Jacknife  both  correlation  matrices  by  removing  one  item  (row  and  corresponding  column) 
at  a  time,  and  extract  the  first  k  (k=I,2,3)  eigenvalues  of  all  the  (n-  l)x(n-l)  submatrices. 

(5)  The  corresponding  eigenvalues  of  the  observed  and  expected  submatrices  are  matched  and 

k  ratios  (k=:  1,2,3)  of  the  form:  • 

RLJj'  =  Xj|.'(observed)/  ^^'(expccted)  (8) 

aie  calculated  for  each  item  (i=l...n). 

(6)  The  n  ratios  in  each  of  the  k  sets  are  rank  ordered,  SWGs  are  calculated  (Wainer  and 
Schacht,  1978),  and  the  largest  SWGs,  MAX(Zjjj),  are  identified. 

(7)  Using  information  (Mean,  SD,  test  of  normality,  etc.)  from  the  distribution  of  the  SWGs 
based  on  the  first  .set  of  matched  eigenvalues  determine  T(Z|),  the  critical  threshold  for 
detecting  unusually  wide  gaps  (supposedly  distinguishing  between  items  from  the 
primary  and  contaminating  pools). 

(8)  Compare  MAXlz^.)  and  MAX(Zj.).  the  largest  SWGs  based  on  the  second  and  third  set  ^ 
of  matched  eigenvalues,  with  T(Zj)  the  critical  rejection  threshold. 
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(9)  If  MAXfZj.)  and/or  MAX(Zj.)  >  T{Zj),  i.e.  there  is  a  significant  gap  in  either  distribution 
■*  of  ratios,  eliminate  those  items  which  are  located  above  the  significant  gapKs)®. 

^  ( 1 0)  Let  m  j  denote  the  number  of  items  eliminated  (m  j  >  0)  after  this  first  pass  through  the 

data.  Repeat  stages  4-9  with  the  reduced  (n-mjHn-mj)  correlation  matrices.  This 
second  analysis  may  lead  to  the  elimination  of  additional  (say  m^)  items.  Repeat  the 
procedure  with  the  remaining  items,  and  stop  when  the  test  (step  8)  fails  to  detect  items  to 
be  rejected. 

AN  EMPIRICAL  STUDY  OF  RMPA 

Method 

In  this  section  we  report  results  of  an  empirical  study  designed  to  test  RMPA.  Like  most  other 
studies  in  this  area  we  simulated  artificial  test  results  by  combining  real  item  parameters  and  a 
set  of  reascMiable  assumptions  regarding  the  distribution  of  abilities  in  the  population  of  test 
takers.  For  the  purpose  of  this  study  we  contaminated  a  large  unidimensional  pool  by  (various 
{MX)pottions  of)  responses  generated  by  a  second  (nuisance)  ability  correlated  (at  various 
*'  levels)  with  the  first.  The  efficiency  of  the  RMPA  was  assessed  by  its  ability  to  identify 
correctly  the  contaminating  items  and,  consequently,  partition  the  test  into  its  two  basic 
•  components. 

We  expect  this  procedure  to  be  most  efficient  in  cases  of  approximate  unidimensionality.  In 
other  words,  it  should  detect  accurately  relatively  low  levels  of  contamination,  but  not  mixtures 
of  two  (equal)  abilities.  We  also  predict  that  the  accuracy  of  the  detection  will  be  inversely 
related  to  the  correlation  between  the  two  abilities  involved. 

Design 

We  generated  20  distinct  "artificial  tests".  The  following  characteristics  were  fixed  for  all  the 
tests: 

^  n  =  test  length  =  80  items; 

N  =:  sample  size  =  2(XX}  examinees; 
t  =  number  of  abilities  =  2. 
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The  following  variables  were  manipulated  across  tests; 

p  5=  proportion  of  contaminating  items  =  0%,  10%,  25%  or  50%  (p=0%  is  a  a  strictly, 
uncontaminated,  unidimensionai  test  and  the  other  three  cases  represent  tow,  medium 
and  high  levels  of  contamination); 

r  =  the  correlation  between  0|  and  Sj,  the  two  abilities  =  0.0, 0.5, 0.7  (the  three  values  are 
approximately  equally  spaced  in  terms  of  r^). 

Replications:  All  combinations  of  p  and  r  were  replicated  twice  (i.e.  with  different  seeds  for 
the  generation  of  the  abilities,  and  different  item  parameters).  In  the  sequel  the  two  replications 
are  labeled  "B”  and  "R". 


This  design  is  sununarized  in  the  10  cells  of  the  following  table.  With  the  exception  of  the 
control  condition  (p=0,  r=0),  this  can  be  viewed  as  a  factorial  crossing  of  two  independent 
variables  repeated  twice. 


r=coiTelation 
between  abilities 

p=%  of  contamination 

0  10  25  50 

0.0 

X 

X 

X  X 

0.5 

- 

X 

X  X 

0.7 

X 

X  X 

Item  Parameters 

The  items  for  half  the  tests  (replication  "R”)  were  randomly  selected  from  the  item  bank  of  a 
test  of  English  as  a  Foreign  Language  (EFL).  This  test  was  developed  and  is  routinely  used 
by  the  National  Institute  for  Testing  and  Evaluation  (NITE)  as  part  of  the  Psychometric 
Entrance  Test  (PET)  which  is  administered  to  all  applicants  to  universities  in  Israel.  The  item 
parameters  were  estimated  under  the  three  parameter  logistic  model  (Equation  2)  using 
responses  from  approximately  7,000  examinees  who  took  the  test  in  1988.  The  estimation 
was  performed  using  the  NITEST  parameter  estimation  software  (Cohen  &  Bodner,  1989). 
These  parameter  estimates  for  the  n=80  items  will  henceforth  be  referred  to  as  "true 
parameters".  They  are  listed  in  Appendix  1. 
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The  items  for  the  other  tests  (replication  "B")  were  generated  artificially,  according  to  some 
distributional  assumptions:  The  discrimination  parameters  (a's)  were  sampled  from  a  normal 
distribution  with  a  mean  of  1.1  and  a  s.d.  of  0.3;  The  difficulty  parameters  (b’s)  were 
obtained  from  a  normal  distribution  with  a  mean  of  0  and  a  s.d.  of  0.8;  The  pseudo-guessing 
parameters  (c's)  are  taken  from  a  uniform  distribution  over  the  range  0. 1  -  0.3.  The  values  of 
the  three  parameters  were  sampled,  from  the  respective  sources,  independently.  The  "true 
parameters"  of  the  "B"  tests  are  listed  in  Appendix  2. 

Table  1  summarizes  the  information  regarding  the  two  sets  of  true  parameters.  The  two  tests 
are  equally  difficult,  but  vary  with  respect  to  other  aspects.  The  discrimination  parameters  of 
the  real  items  ("R")  have  a  higher  mean  and  variance  (m^=1.33  and  s^=O.S  1)  than  the  artificial 
ones  ("B")  (m  =1.12  and  s„=0.25).  On  the  average,  it  is  easier  to  guess  in  the  artificial  test 
(m^=0.2  vs.  0.16).  Finally,  whereas  the  parameters  of  the  artificial  items  are  uncorrelated  (by 
design),  the  values  of  the  EFL  items  parameters  are  moderately  correlated. 


Insert  Table  I  about  here 


Atiililks 

All  samples  include  N=2(X)0  simulated  "respondents".  First  we  generated  four  mutually 
uncorrelated  sets  of  abilities  (T,  A|,  A^  and  A^):  We  sampled  8(XX)  independent  observations 
from  the  standard  (0,1)  normal  distribution  and  randomly  assigned  them  to  the  four  sets. 
Correlated  abilities  were  generated  by  calculating: 

T(r)  =  r  T  +  Vl  -r2  A,  (H) 

where  Aj  stand  for  Aj,  or  Aj,  and  r  is  the  desired  correlation  (0.0, 0.5, 0.7)  between  the 
new  set  of  abilities,  T(r),  and  the  reference  set,  T.  Thus  T(0),  T(.5),  T(.7)  are  sets  of  N=2(X)0 
normally  distributed  abilities  which  correlate  0.0. 0.5  and  0.7,  respectively,  with  T. 

Responses 

Four  sets  of  unidimensional  response  vectors  were  generated.  Each  set  was  simulated  with  a 
different  set  of  abilities  |T,  T(0),T(.5)  orT(.7)),  and  all  responses  were  generated  with  the 
"true"  item  parameters.  The  response  vectors  were  simulated  with  the  NfTECAT  software 
package  (Cohen,  Bodner  &  Ronen,  1989),  which  implements  the  process  described  by 
Drasgow  and  Lissak  (1983). 
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The  vectors  generated  with  the  T  abilities  are  considered  the  "original"  responses  based  on  the 
dominant  ability.  Contaminated  responses  were  obtained  by  replacing  the  original  responses 
on  p%  of  the  items  (randomly  selected)  with  the  corresponding  responses  generated  by  one  of  ^ 
the  other  samples  of  abilities.  Note  that  for  the  case  of  r=0  this  procedure  simulates  a  two- 
dimensional  "noncompensatory"  model  (e.g.  Ackerman.  1989,  Sympson,  1978),  whereas  the 
other  cases  (r  >  0)  simulate  "compensatory"  models  (e.g.  Ackerman,  1989,  Reckase,  1985). 

Parameter  estimation 

In  each  of  the  artificial  tests  the  three  parameters  of  the  n=80  items  were  estimated  with  the 
NITEST  program  (Cohen  &  Bodner,  1989).  These  are  the  various  sets  of  "estimated 
parameters",  to  be  used  in  the  generation  of  the  expected  correlations. 

Consistent  with  the  massive  literature  on  this  topic  (e.g.  Dorans  &  Kingston.  1985;  Miller  & 
Oshima,  1992;  Oshima  &  Miller,  1992),  we  found  that  the  estimates  of  the  b's  and  c's  were 
not  affected  by  the  contamination.  However,  the  estimates  of  the  a’s  (the  discrimination 
parameters)  are  sensitive  to  the  level  of  contamination.  Appendix  3  presents  the  mean 
estimates  of  the  a  parameters  for  items  loaded  on  the  dominant  and  nuisance  trait.  The  pattern 
and  magnitude  of  the  estimates  is  consistent  with  other  studies  in  the  literature:  The  estimates 
for  items  loaded  on  the  dominant  ability  are  hardly  affected,  whereas  the  discrimination 
measures  of  the  contaminating  items  are  reduced  considerably.  The  magnitude  of  this 
"shrinkage"  is  related  to  the  level  of  contamination  and  the  correlation  between  the  two  factors. 

A  very  similar  pattern  is  observed  when  comparing  communality  estimates  (expected  test-retest  . 
reliabilities  of  the  items).  The  results  of  this  comparison  are  summarized  in  Appendix  4. 

Results 

The  data  were  analyzed  according  to  the  ten  steps  procedure  outlined  in  the  summary  above. 

We  report  the  main  results  according  to  the  same  sequence. 

Standard  MPA 

At  the  conclusion  of  the  third  stage  one  can  perform  the  standard  MPA,  prescribed  by 
Drasgow  and  Lissak  (1983).  Table  2  summarizes  these  results.  The  table  displays  the  first 
three  eigenvalues  of  both  correlation  matrices,  as  well  as  their  ratios. 


Insert  Table  2  about  here 
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There  is  a  clear  and  consistent  pattern  in  the  data  which  can  be  sunnnarized  by  three 
observations: 

(i)  The  first  eigenvalues  are,  practically,  equal  in  the  two  matrices  and  their  ratio  is, 
essentially,  1.  There  are  no  discernible  differences  between  the  18  contaminated  data  sets 
and,  in  this  respect,  they  are  indistinguishable  from  the  two  uncontaminated  tests. 

(ii)  In  all  contaminated  tests,  the  second  eigenvalue  of  the  observed  matrix  is  larger  than  its 
expected  counterpart.  Consequently,  their  ratio  is  greater  than  unity,  as  predicted  by 
Drasgow  &  Lissak  (1983).  The  ratio  is  a  monotonically  increasing  function  of  p,  the  level 
of  contamination,  and  a  monotonically  decreasing  function  of  r,  the  inter-ability 
correlation. 

(iii) The  ratio  of  the  third  pair  of  eigenvalues  is  also  greater  than  one.  In  fact,  in  most  cases  it 
is  greater  than  the  second  ratio.  The  third  ratio  is  not  systematically  related  to  r,  the  inter¬ 
ability  correlation.  However,  it  increases  monotonically  as  a  function  of  p,  the  level  of 
contamination.  The  sharpest  effect  is  obtained  for  highly  (r=0.7)  correlated,  and  the 
weakest  effect  is  found  for  uncorrelated  (r=0.0)  abilities. 

RMPA 

At  the  conclusion  of  the  fifth  stage  one  can  perform  an  informal  RMPA  by  examining  the 
eigenvalues  of  the  jacknifed  parallel  matrices.  Table  3  displays  means,  and  standard 
deviations,  of  the  first  three  eigenvalues  extracted  from  the  jacknifed  submatrices.  All  the 
values  in  the  table  are  based  on  n=80  matrices  of  order  (n-l)=79.  Note  that  the  mean  values 
are  related  to  the  eigenvalues  from  table  2  through  multiplication  by  a  scale  factor  of  n/(n-l)= 
80/79. 


Insert  Table  3  about  here 


Table  4  presents  ratios  of  the  means,  and  the  variances,  of  the  three  Jacknifed  eigenvalues  of 
the  20  tests. 


Insert  Table  4  about  here 


There  is  a  close  correspondence  between  these  mean  ratios  and  the  ratios  presented  in  table  2, 

and  the  same  three  basic  conclusions  apply  here,  as  well.  The  ratios  of  the  variances  follow  a 

similar,  but  not  identical,  pattern: 

(i)  The  variances  of  the  first  eigenvalues  are,  on  the  average,  very  close  to  each  other  and  their 
ratio  is  close  to  unity.  The  only  exceptions  are  the  cases  {r=0,  p=50},  which  represent 
mixtures  of  two  unidimensional  half-tests  involving  uncorrelated  abilities. 

(ii)  In  most  cases  (and  on  the  average)  the  variance  of  the  second  (jacknifed)  eigenvalues  in  the 
observed  matrices  is  higher  than  in  the  expected  one.  The  effect  is  most  pronounced  in  the 
case  of  the  independent  traits  (r=0),  and  for  moderate  or  high  levels  of  contamination 
(p=25  and  50,  respectively). 

(iii)  In  all  20  tests  the  variances  of  the  third  (jacknifed)  eigenvalues  are  substantially  higher  in 
the  observed  matrices.  The  effect  is  much  stronger  than  for  the  second  eigenvalue,  but 
there  is  no  systematic  pattern  of  change  across  levels  and  types  of  contamination. 

Table  5  presents  the  correlations  between  the  matched  jacknifed  eigenvalues  for  the  20  tests. 

Each  correlation  is  based  on  n=80  observations. 

Insert  Table  5  about  here 

The  pattern  of  results  is  clear  and  consistent  with  our  expectations; 

(i)  There  is  a  high  (almost  perfect)  linear  correlation  for  the  first  eigenvalue  in  most  tests.  The 
single  exception  is  the  {Rep=R,  r=0,  p=50}  case,  which  is  a  mixture  of  two  uncorrelated 
(unidimensional)  half-tests. 

(ii)  In  all  cases  of  moderate  and  high  contamination  (p=25  and  50,  respectively)  the 
correlations  based  on  the  second  and  third  eigenvalue  are  low,  or  negative. 

(iii)  In  most  cases  of  low  contamination  (p=10)  the  correlations  based  on  the  second  eigenvalue 
are  high  (almost  like  for  the  first  eigenvalue),  but  the  correlations  based  on  the  third 
eigenvalue  are  always  low,  or  negative. 
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This  pattern  indicates  that,  as  suggested  by  Drasgow  and  Lissak  (1983)  and  others,  the  first 
eigenvalues  of  the  two  parallel  matrices  are  practically  indistinguishable,  across  all  types  and 
levels  of  contamination.  However,  contrary  to  Drasgow  and  Lissak's  speculation,  not  all  the 
differences  between  the  two  data  sets  can  be  detected  by  comparing  the  second  pair  of 
eigenvalues.  The  means,  variances  and  correlations  of  the  jacknifed  values  seem  to  suggest 
that  in  some  cases  of  low  contanunation  (p=0.10)  violations  from  unidimensionality  can  only 
be  detected  by  examining  the  third  pair  of  eigenvalues. 

Rejection  Thresholds 

Table  6  presents  seven  rejection  thresholds  calculated  from  the  distribution  of  the  first  ratio  in 
the  20  tests.  The  first  is,  simply,  the  2.2S  value  proposed  by  Wainer  and  Schacht  (1978). 
The  other  six  are  obtained  by  crossing  two  confidence  levels  (95%  and  99%)  with  three  rules 
of  detection  —  an  empirical  value,  a  value  calculated  by  the  "tight"  (i.e.  assuming 
unidimodality  and  symmetry)  Chebyshev  inequality,  and  a  value  derived  from  the 
unconstrained  Chebyshev  inequality. 


Insert  Table  6  about  here 


In  ail  tests,  and  for  both  confidence  levels,  the  empirical  percentile  is  more  liberal  than  the 
corresponding  Chebyshev  bounds.  Thus,  the  three  rules  can  be  ranked,  from  the  most  to  the 
least  conservative,  identically  for  all  tests  and  for  both  levels  of  confidence: 

Unconstrained  Chebyshev  >  Constrained  Chebyshev  >  Empirical 

Tte  2.2S  value  is,  in  all  cases,  more  extreme  than  the  empirical  95th  percentile,  but  smaller 
than  all  the  Chebyshev  bounds.  In  most  cases  (13/20)  the  99th  empirical  percentile  is  above 
2.25.  One  remarkable  and  reassuring  aspect  of  this  table  is  the  relatively  low  variance  of  the 
bounds  across  the  various  conditions  and  replications.  This  indicates  that  the  ratio  of  the  first 
pair  of  jacknifed  eigenvalues  has  a  relatively  stable  distribution  across  the  levels  and  types  of 
contamination. 

To  further  examine  the  performance  of  the  rejection  thresholds  we  calculated  the  proportion  of 
SWGs  which  were  found  to  be  higher  than  the  threshold,  in  the  various  tests.  The  results  for 
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the  18  contaminated  tests  are  summarized  in  Appendix  5.  The  proportions  are  summarized  as 
a  function  of  the  eigenvalue  examined  (first,  second  or  third),  the  level  of  contamination  and 
the  inter-trait  correlation.  The  overall  trend  is  for  the  number  of  unusually  large  gaps  to 
increase  monotonically  as  a  function  of  the  eigenvalue  (it  is  lowest  for  the  first  and  highest  for 
the  third),  and  the  level  of  contamination,  and  decrease  monotonically  with  r,  the  inter-ability 
correlation.  The  actual  rates  of  change  vary  from  one  threshold  to  another. 

The  most  important  issue,  from  a  practical  point  of  view,  is  to  choose  the  "best"  threshold  for 
detection  of  wide  gaps.  To  address  this  issue  we  focus  on  the  performance  of  the  various 
indices  in  the  uncontaminated  (p=0)  case.  Table  7  displays  the  proportion  of  SWGs  exceeding 
the  various  indices  for  the  three  ratios.  Since  this  is  a  strictly  unidimensional  test,  we  expect 
this  proportion  to  be  invariant  for  ail  three  ratios  and  not  to  exceed  its  nominal  confidence  level 
(95%  or  99%).  Clearly,  2.25  and  the  empirical  percentiles  fail  the  invariance  requirement  and 
the  95%  constrained  Chebyshev  bound  is  too  liberal  for  the  third  ratio.  In  light  of  these  results 
we  conclude  that  is  best  to  identify  as  "unusually  wide  gaps"  those  values  that  exceed  the  95% 
unconstrained,  or  the  constrained  99%  Chebyshev  bounds.  We  will  focus  primarily  on 
rejections  with  99%  confidence.  However,  for  completeness  sake,  we  will  report  in  the  sequel 
results  according  to  all  the  seven  thresholds. 


Insert  Table  7  about  here 


Partilwn  of  ihg  Tgsts 

Tables  8a  -  8c  list  the  maximal  SWGs  observed  in  the  distributions  of  the  three  ratios  for  each 
test.  The  tables  also  display  the  pattern  of  significance  achieved  by  this  maximal  SWG,  and  its 
location.  The  columns  labeled  "significance"  simply  count  how  many  (of  the  increasingly 
stringent)  thresholds  were  exceeded  in  each  family  of  tests.  The  fixed  2.25  criterion  is  either 
surpassed  (1  in  the  table)  or  not  (0).  In  the  95%  and  99%  columns,  a  1  indicates  that  the 
observed  value  is  greater  than  the  empirical  percentile  but  lower  than  both  Chebyshev  bounds; 
a  value  of  2  describes  a  situation  where  the  actual  value  is  greater  than  the  constrained  (but 
smaller  than  the  unconstrained)  Chebyshev  bound,  and  a  value  of  3  denotes  a  case  where  the 
maximal  gap  is  larger  than  the  most  severe  rejection  rule.  Our  previous  results  (see  table  7) 
dictate  to  interpret  as  "significant"  values  of  2  (at  99%),  or  values  of  3  (at  95%). 
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The  location  of  the  gii|i  is  described  by  reporting  the  number  of  items  above,  and  below,  it. 
Recall  that  according  to  the  logic  of  RMPA  the  contaminating  items  should  have  lower  (i.e. 
closer  to  unity)  ratios.  We  rank  ordered  the  ratios  in  ascending  order,  so  these  items  are 
expected  to  cluster  "above"  the  gap.  As  a  rule,  we  expect  the  proportion  of  item  above  the  gap 
to  match,  approximately,  the  proportion  of  contamination  in  the  specific  test.  Since  decisions 
about  rejection  can  be  based  on  the  second  and/or  third  eigenvalue,  we  summarize  in  table  9 
the  pattern  of  results  for  each  test  across  all  three  ratios. 

We  reject  the  null  hypothesis  of  unidimensionality  if: 

(1)  The  number  of  items  "above  the  gap"  <  n/2  AND 

(2)  The  Maximal  SWG  of  the  second  AND/OR  the  third  ratio  is  greater  than  the  designated 
rejection  threshold. 

We  examine  three  rejection  rules  with  decreasing  levels  of  conservatism:  (1)  99%  according  to 
an  unconstrained  Chebyshev  inequality,  (2)  99%  according  to  a  constrained  Chebyshev 
inequality,  and  (3)  95%  according  to  the  unconstrained  Chebyshev  inequality. 


Insert  Tables  8a  -  8c  and  9  about  here 


As  expected,  there  are  no  significant  gaps  in  the  distribution  of  the  first  ratio  but,  in  most  tests, 
the  largest  SWG  in  the  distribution  of  the  second  and/or  third  ratio  is  significant.  We  examine 
these  significant  gaps  according  to  the  three  valid  rejection  thresholds: 

The  most  stringent  approach  requires  a  SWG  to  exceed  the  99%  threshold  derived  from  a 
regular  Chebyshev  inequality.  Seven  tests  have  gaps  larger  than  this  threshold  (three  in 
the  distribution  of  the  second  ratio,  two  in  the  distribution  of  the  third  and  one  in  both).  Five 
of  these  tests  have  low  (p=10)  level  of  contamination,  one  is  moderately  (p=25)  and  the  other 
is  highly  (p=SO)  contaminated. 

In  seven  of  the  remaining  tests  the  Max(SWG)  exceeds  the  99%  threshold  derived  from  a 
constrained  (unimodality  +  symmetry)  Chebyshev  inequality.  One  is  uncontaminated  (p=0), 
one  is  slightly  (p=10),  three  are  moderately  (25%)  and  two  are  highly  (p=50)  contaminated. 
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All  the  other  six  tests  reach  significance  according  to  an  unconstrained  95%  Chebychev  bound. 
This  group  includes  one  uncontaminated  (p=0)  test  as  well  as  two  moderately  (25%)  and  three 
highly  (p=50)  contaminated  cases. 

All  six  cases  with  low  (p=10)  contamination  are  significant  at  the  99%  level  (five  of  them  by 
the  most  severe  criterion).  In  all  six  cases  the  gap  separates  the  top  10%  items  from  the  bottom 
90%.  It  appears  that  the  procedure  works  well  for  this  type  of  contamination. 

Only  three  of  the  highly  contaminated  tests  (p=50)  are  significant  at  99%.  More  important, 
however,  is  the  fact  that  in  alt  six  tests  the  widest  gap  is  located  at  the  bottom  of  the 
distribution.  Although  the  numbers  vary  slightly  across  tests,  the  proportion  of  items  above 
the  gap  is  always  greater  than  80%.  Clearly,  the  gap  test  does  not  work  well  for  a  mixture  of 
two  half  tests. 

The  pattern  of  results  is  slightly  more  complex  in  the  case  of  moderate  (p=25)  contamination, 
and  it  depends  on  the  level  of  the  inter-ability  correlation:  For  both  tests  with  uncorrelated 
(r=0)  abilities,  and  one  of  the  tests  with  moderately  correlated  (r=0.5)  abilities,  the  significant 
gap  (99%)  in  the  distribution  of  the  second  ratio  separates  the  upper  25%  items  from  the  rest  of 
the  test.  In  the  other  test  with  r=0.S  the  gap  between  the  top  25%  of  the  items  and  the  lower 
75%  is  significant  at  the  95%  level.  Finally,  for  the  tests  involving  highly  correlated  abilities 
(p=0.7),  the  maxima]  gap  is  located  at  the  lower  end  of  the  distribution  (69  and  72  items  above 
the  gap).  In  both  cases  the  second  largest  gap  distinguishes  between  the  (most)  contaminating 
items  and  the  original  ones.  Thus,  the  gap  test  operates  well  only  for  cases  with  low  inter¬ 
ability  correlations. 

To  summarize,  RMPA  found  a  significant  gap  in  the  distribution  of  the  ratios  of  matched 
eigenvalues  in  al  ;  the  tests  examined.  In  14  tests  the  gap  was  significant  at  99%  and  in  the 
othc*  six  at  95%.  A  significant  gap  located  in  the  upper  half  of  the  distribution  (i.e.  with  fewer 
items  above  the  gap  than  bellow  it)  is  taken  as  a  strong  indication  of  violation  of 
unidimensionality  and  prescribes  elimination  of  all  items  above  the  gap.  The  ten  tests 
identified  by  this  criterion  include  all  those  with  low  contamination  (p=10),  as  well  as  the 
moderately  contaminated  ones  (p=25),  with  moderate  level  of  inter-ability  correlation  (r  <0.7). 

In  the  sequel  we  focus  only  on  these  10  shortened  tests.  To  facilitate  interpretation  of  the 
results,  we  attach  plots  of  the  10  relevant  distributions  of  standardized  weighted  gaps.  The 
original  ( l-p)n  items  are  plotted  as  "*''s  and  the  pn  contaminating  items  are  plotted  as  "C'  s. 
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Note  that  in  all  plots: 

(i)  the  contaminating  items  are  clustered  at  one  end  of  the  distribution,  and 

(ii)  there  is  an  unusually  large  gap  separating  this  cluster  from  the  bulk  of  the  items.  This  gap 
can  be  detected  in  the  raw  gaps,  but  it  is  more  pronounced  in  the  standardized  weighted 
form. 


Insert  Figures  1-10  about  here 


The  quality  of  the  technique  is  assessed  by  its  ability  to  detect  the  contaminating  items  and 
remove  them,  while  retaining  the  original  ones.  Table  10  summarizes  this  analysis  for  the  10 
short  tests.  For  each  one  we  report  the  hit  rate  (i.e.  contaminating  items  rejected  conectly) 
and  the  false  alarm  rate  (i.e.  original  items  rejected  incorrectly).  The  figures  are  very 
impressive  —  for  ail  the  tests  with  p-10%,  the  hit  rate  is  100%  and  for  the  tests  with  p=25% 
it  is  95%.  Both  figures  are  accompanied  by  false  alarm  rates  close  to  0.  This  impression  can 
be  also  verified  in  their  ROC  curves  (e.g.  Green  &  Swets,  1973).  These  curves  plot  the  hit 
rate  against  the  false  alarm  rate  for  20  equally  spaced  rejection  thresholds.  Each  figure 
includes  a  curve  based  on  the  ratio  of  the  first  pair  of  eigenvalues  and  one  based  on  the  ratio  of 
the  second  or  third  (the  one  that  reached  significance  in  that  particular  test).  In  all  ten  cases  the 
latter  curve  stochastically  dominates  the  former.  Furthermore,  at  practically  all  points  the 
procedure  does  a  perfect  job  of  detecting  the  contaminating  items. 


Insert  Table  10  and  Figures  11-20  about  here 


Bfi-sxaminaUon  of  the  shortened  tests 

Having  shortened  10  tests  according  to  the  results  of  the  initial  RMPA  we  repeated  steps  4  -  9 
of  the  procedure.  The  second  iteration  verifies  the  unidimensionality  of  the  shortened  tests:  If 
the  first  stage  is  successful  in  removing  all  sources  of  contamination,  we  do  not  expect  to 
detect  any  significant  gaps  in  this  second  round. 

Tables  1 1  and  12  report  the  results  of  the  MPA  and  the  RMPA  of  the  shortened  tests.  A  quick 
comparison  with  tables  2  and  4  (summarizing  the  same  results  for  the  original  full  tests) 
reveals  that  all  major  sources  of  multidimensionality  were  eliminated.  The  ratios  of  the  second 
eigenvalues,  and  the  ratios  of  their  variances,  are  close  to  unity  (We  assume  that  a  heuristic 
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MPA  wo’ild  also  declare  all  these  tests  unidimensional).  The  third  ratios  are  somewhat  higher 
but  are,  considerably,  lower  than  those  of  the  original  tests. 


i 

Insert  Tables  1 1  and  12  about  here 


The  SWGs  of  the  remaining  items  were  calculated,  new  rejection  thresholds  were  derived,  and 
the  gap  test  was  applied  again  The  results  are  presented  in  Tables  1 3a- 1 3c  (parallel  in 
structure  and  notation  to  tables  8a-8c). 


Insert  Tables  1 3a  -  13c  about  here 


As  expected,  none  of  the  ratios  based  on  the  first  pair  of  eigenvalues  is  significant  (according 
to  the  99%  Chebyshev  bounds).  We  found  significant  gaps  in  the  distribution  of  the  second 
ratio  for  four  tests,  and  three  of  them  also  revealed  significant  gaps  in  the  third  ratio. 

However  vith  one  exception,  the  significant  gaps  are  in  the  lower  tail  of  the  distribution. 
Therefore,  they  are  not  indicative  of  violations  of  unidimensionality.  The  only  exception  was 
the  { Rep=B  r=0.7  p=l01  test.  In  this  case  the  second  iteration  of  the  RMPA  prescribes 
removal  of  five  additional  items.  All  contaminated  items  were  successfully  detected  by  the  first 
iteration  so  these  are  five  "false  alarms".  The  final  test  consists  of  66  unidimensional  items 
(instead  of  72). 
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SUMMARY 

1  The  goal  of  the  current  research  was  to  develop  a  practical,  yet  theoretically  sound  and 
computationally  feasible,  tool  for  testing  the  global  dimensionality  of  large  item  pools  and 
eliminating  items  which  cause  violations  of  the  pool's  unidimensionality.  Both  goals  are 
attained  in  the  unified  framework  of  RMPA. 

MPA  was  developed  by  Drasgow  and  Lissak  (1983)  as  an  approximate  method  for  testing  the 
unidimensionality  of  item  pools.  It  relies  on  a  heuristic  comparison  of  a  statistic  (the  second 
eigenvalue)  derived  from  the  matrix  of  items'  intercorrelations  and  the  corresponding  value 
extracted  from  a  "parallel"  matrix  generated  by  a  unidimensional,  and  locally  independent, 
model  (in  our  case  the  three  parameter  logistic  model). 

RMPA  is  based  on  a  similar  comparative  logic,  but  improves  upon  MPA  in  several  ways; 

(1)  It  alleviates  some  minor  technical  limitations,  through  tlK  use  of  expected  correlations 
under  unidimensionality; 

(2)  It  implements  a  formal  test  for  comparing  the  observed  data  set  with  its  parallel  (and 
unidimensional)  counterpart. 

(3)  Contingent  upon  the  results  of  this  test,  it  provides  a  method  for  identifying  and 
eliminating  items  which  violate  the  test's  unidimensionality. 

* 

The  testing  and  elimination  procedures  are  based  on  the  "remove  one  item  at  a  time"  principle. 
This  methodology  allows  one  to  assess  the  contribution  of  each  item  to  the 
test's  eigenvalues.  Furthermore,  one  can  determine  the  variance  and  distribution  of  these 
valiKs  and  analyze  the  differential  impact  of  any  given  item  in  the  observed  and  parallel 
matrices.  Items  which  have  a  "significantly"  larger  impact  in  the  observed  data  set  violate 
unidimensionality. 

The  detection  of  these  items  relies  on  a  conservative  version  of  Wainer  &  Schacht's  (1978) 
"gapping"  test.  The  largest  (first)  eigenvalues  of  the  observed  and  expected  matrices  are 
practically  identical  in  all  cases,  regardless  of  the  level  of  correlation  between  the  two  abilities 
and  the  degree  of  contamination.  Therefore,  we  used  the  distribution  of  their  ratio  to  determine 
rejection  thresholds  for  the  ratio  of  the  second  and  third  eigenvalues.  These  thresholds  are 
based  on  conservative  Chebyshev  bounds,  and  are  specifically  tailored  to  each  test. 


f 
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RMPA  was  tested  in  several  simulations  of  unidimensional  item  pools  which  were 
contaminated  by  various  proportions  of  items  loaded  on  a  secondary  nuisance  ability.  The 
method  was  highly  successful  in  identifying  low  (10%)  levels  of  departure  from 
unidinrcnsionality,  and  in  detecting  moderate  (2S%)  deviations  from  unidimensionality  when 
the  abilities  were  not  highly  (r  <  0.7)  correlated.  In  these  cases  over  90%  of  the  contaminating 
items  were  identified  and  less  than  1%  of  the  original  items  were  eliminated  erroneously.  The 
procedure  failed,  and  should  not  be  applied,  in  tests  which  are  equal  mixtures  (50%)  of  two 
abilities. 

We  conclude  by  pointing  out  that  the  logic  of  MPA  and  RMPA  can  be  generalized  to  other 
statistics  of  closeness  between  the  two  data  sets.  For  example,  it  might  be  interesting  to  apply 
it  to  indices  derived  from  non  linear  factor  analysis  (e.s.  McDonald,  1982). 
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FOOTNOTES 


(1)  Strictly  speaking  "jacknifmg"  refers  to  an  analysis  in  which  observations  (i.e.  respondents) 
are  eliminated  one  at  a  time  from  tte  sample.  In  this  case,  we  eliminate  variables  (items)  in 

*  a  similar  fashion.  Several  item  analysis  computer  programs  use  a  similar  approach  in  order 
to  identify  subscales  with  maximal  reliability. 

(2)  Ip  the  sample  influence  function  (Devlin,  Gnanadesikan  and  Kettenring,  1975;  Hampel, 
1974)  of  parameter,  T,  is  given  by: 

I.  =  (n-1)(T-Tj). 

where  n  is  the  number  of  items,  and  T .  is  an  estimate  of  the  parameter  T  obtained  after  the 
elimination  of  item  i.  Note  that  I.  is,  simply,  a  linear  transformation  of  T  j. 

(3)  The  term  "conservative"  is  used  here  according  to  the  standard  convention  in  statistical 
inference,  i.e.  a  procedure  is  more  conservative  than  its  competitor  if  it  invokes  a  more 
stringent  criterion  in  rejecting  the  null  hypothesis. 

(4)  Strictly  speaking,  Chebyshev  inequality  requires  knowledge  of  the  parameters  (mean  and 
variance)  of  the  population  of  interest.  However,  Saw,  Yang  and  Mo  ( 1984)  have  shown 

,  that  sample  estimates  of  these  parameters  can  be  used,  with  very  little  loss  of  precision,  in 
moderately  large  samples. 

(5)  Occasionally  a  large  (and  significant)  gap  will  be  detected  in  the  lower  tail  of  the 
distribution,  i.e.  separating  the  bulk  of  the  data  from  a  minority  of  items  with  unusual  low 
ratios  of  observed/expected  jacknifed  eigenvalues.  Clearly,  these  cases  are  not  relevant  for 
our  goal. 

(6)  Since  the  procedure  is  data  driven,  we  opt  not  to  use  the  thresholds  values  employed  in  the 
first  stage.  Thus,  when  analyzing  a  test  consisting  of  (n  -  mj)  items  one  should  obtain  the 
same  results,  and  reach  the  same  conclusions,  whether  it  is  treated  as  "an  original"  or  "a 
reduced"  test. 
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Table  1: 

Means,  standard  deviations  and  correlations  of  the  two  sets  of  item  parameters  » 


Rep=B 

Parameter  n  Mean  Std.  Dev. 

Rep=R 

Parameter  n  Mean  Std.  Dev. 

a  80  1.123  0.245 

b  80  0.172  0.873 

c  80  0.202  0.057 

a  80  1.328  0.511 

b  80  -0.026  0.985 

c  80  0.161  0.098 

Correlations 

a  b  c 

Correlations 

a  b  c 
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Table  2: 

Modified  Parallel  Analysis  {MPA}  of  20  tests: 
The  first  three  eigenvalues  for  the  observed  and 
expected  matrices,  and  their  ratios 


Rep 

r 

P 

Exp 

Eigenvalue  1 

Obs  Obs/Exp 

Eigenvalue  2 

Exp  Obs  Obs/Exp 

Eigenvalue  3 

Exp  Obs  Obs/Exp 

B 

0 

24.34 

25.10 

1.03 

1.79 

1.78 

0.99 

0.17 

0.67 

4.02 

B 

0.0 

10 

22.03 

22.71 

1.03 

1.62 

2.55 

1.58 

0.17 

1.66 

9.72 

B 

0.0 

25 

18.15 

18.80 

1.04 

1.43 

5.59 

3.92 

0.15 

1.53 

10.17 

B 

0.0 

50 

11.93 

12.58 

1.05 

0.61 

12.01 

19.77 

0.08 

0.85 

10.35 

B 

0.5 

10 

22.72 

23.55 

1.04 

1.66 

1.78 

1.07 

0.18 

1.66 

9.13 

B 

0.5 

25 

17.93 

20.77 

1.04 

1.46 

3.87 

2.65 

0.16 

1.54 

9.75 

B 

0.5 

50 

17.76 

18.71 

1.05 

0.92 

6.05 

6.60 

0.07 

1.07 

14.38 

B 

0.7 

10 

23.31 

24.16 

1.04 

1.66 

1.73 

1.04 

0.18 

1.26 

7.09 

B 

0.7 

25 

21.27 

22.13 

1.04 

1.46 

2.30 

1.58 

0.15 

1.55 

10.51 

B 

0.7 

50 

19.91 

20.82 

1.05 

1.18 

3.67 

3.11 

0.09 

1.34 

14.26 

R 

0 

26.20 

26.23 

1.00 

3.47 

3.22 

0.93 

0.36 

0.63 

1.78 

R 

0.0 

10 

23.59 

23.84 

1.01 

2.84 

2.86 

1.01 

0.25 

2.57 

10.20 

R 

0.0 

25 

19.57 

19.81 

1.01 

2.38 

6.20 

2.60 

0.20 

2.33 

11.91 

R 

0.0 

50 

12.23 

12.90 

1.05 

1.45 

12.20 

8.39 

0.15 

1.77 

11.53 

R 

0.5 

10 

23.68 

24.14 

1.02 

2.76 

2.76 

1.00 

0.28 

1.90 

6.69 

R 

0.5 

25 

21.45 

21.90 

1.02 

2.48 

4.31 

1.74 

0.25 

2.45 

9.70 

R 

0.5 

50 

19.30 

19.88 

1.03 

1.79 

6.35 

3.55 

0.13 

1.93 

14.89 

R 

0.7 

10 

24.45 

24.86 

1.02 

2.88 

2.87 

1.00 

0.30 

1.18 

4.01 

R 

0.7 

25 

22.91 

23.30 

1.02 

2.64 

3.16 

1.20 

0.23 

2.28 

10.09 

R 

0.7 

50 

21.78 

22.25 

1.02 

2.32 

3.89 

1.68 

0.17 

2.39 

14.17 

Notes: 

All  results  based  on  n=80  items  and  N=2000  respondents. 
Exp  =  Derived  from  matrix  of  expected  correlations 
Obs  =  Derived  from  matrix  of  observed  correlations. 
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Table  3: 


Revised  Modified  Parallel  Analysis  (RMPA)  of  20  tests: 

Means  and  standard  deviations  of  the  first  three  eigenvalues  of  the  jacknifed 
submatrices  (observed  and  expected)  _ _ _ 


Rep 

r 

P 

Source 

Eigenvalue  1 

Mean  SD 

Eigenvalue  2 

Mean  SD 

Eigenvalue  3 

Mean  SD 

B 

0 

Exp 

24.036 

0.122 

1.764 

0.029 

0  165 

0.004 

Obs 

24.786 

0.118 

i.756 

0.027 

0.664 

0.009 

B 

0.0 

10 

Exp 

21.752 

0.145 

1,598 

0.026 

0.168 

0.004 

Obs 

22.429 

0.147 

2.515 

0.103 

1.637 

0.027 

B 

0,0 

25 

Exp 

17.920 

0.163 

1.409 

0.027 

0.148 

0,004 

Obs 

18.563 

0.169 

5.519 

0.143 

1.507 

0.027 

B 

0.0 

50 

Exp 

11.778 

0.136 

0.600 

0.013 

0.081 

0.002 

Obs 

12.432 

0.158 

1 1 .842 

0.165 

0.842 

0.016 

B 

0.5 

10 

Exp 

22.437 

0.133 

1.640 

0,027 

0.179 

0.004 

Obs 

23.255 

0.128 

1.769 

0.036 

1.630 

0,038 

B 

0.5 

25 

Exp 

19.685 

0.136 

1.443 

0.026 

0.156 

0.004 

Obs 

20.507 

0.128 

3.818 

0,084 

1.525 

0.027 

B 

0.5 

50 

Exp 

17.536 

0.089 

0.906 

0.014 

0.073 

0.002 

Obs 

18.472 

0.085 

5.972 

0.035 

1.053 

0.019 

B 

0.7 

10 

Exp 

23.023 

0.125 

1.640 

0.026 

0.175 

0.004 

Obs 

23.858 

0.119 

1.713 

0.025 

1.241 

0  043 

B 

0.7 

25 

Exp 

21.008 

0.126 

1.437 

0.024 

0.145 

0.003 

Obs 

21,850 

0.122 

2.267 

0.048 

1.526 

0.027 

B 

0.7 

50 

Exp 

19.660 

0.103 

1.164 

0.019 

0,093 

0002 

Obs 

20.564 

0.101 

3.625 

0,025 

1.326 

0.021 

R 

. 

0 

Exp 

25.874 

0.128 

3.424 

0.040 

0,351 

0.006 

Obs 

25.899 

0.132 

3.178 

0.036 

0.628 

0.007 

R 

0.0 

10 

Exp 

23.292 

0.155 

2.808 

0.040 

0.249 

0.006 

Obs 

23.547 

0.160 

2.829 

0035 

2.531 

0.113 

4 

R 

0.0 

25 

Exp 

19.330 

0.174 

2.354 

0.039 

0.193 

0.005 

Obs 

19.561 

0.182 

6.121 

0.149 

2.301 

0.035 

R 

0.0 

50 

Exp 

12,078 

0.142 

1.436 

0.031 

0.152 

0.004 

Obs 

12.735 

0.187 

12.045 

0.175 

1.751 

0.037 

R 

0.5 

10 

Exp 

23.389 

0.141 

2,726 

0.038 

0.281 

0.006 

Obis 

23.835 

0.138 

2.726 

0.034 

1.876 

0.078 

R 

0.5 

25 

Exp 

21.181 

0.152 

2,449 

0,038 

0.249 

0.006 

Obs 

21.629 

0.143 

4.251 

0.084 

2.415 

0.035 

R 

0.5 

50 

Exp 

19.063 

0.107 

1.763 

0.023 

0.128 

0.004 

Obs 

19.636 

0.101 

6.263 

0.037 

1.906 

0,025 

R 

0.7 

10 

Exp 

24.145 

0.136 

2.839 

0.038 

0.291 

0  006 

Obs 

24.552 

0.132 

2.835 

0.033 

1.166 

0.043 

R 

0,7 

25 

Exp 

22.627 

0.138 

2.605 

0.037 

0.223 

0.005 

Obs 

23.006 

0.132 

3.123 

0.040 

2.252 

0.033 

R 

0.7 

50 

Exp 

21.504 

0.118 

2.292 

0.030 

0.166 

0.004 

- 

Obs 

21.970 

0.116 

3.843 

0.027 

2.357 

0.030 

Notes: 

All  results  based  on  n=80  items  and  N=2000  respondents. 
Exp  =  Derived  from  matrix  of  expected  correlations 
Obs  =  Derived  from  matrix  of  observed  correlations. 
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Table  4: 

^  Revised  Modified  Parallel  Analysis  (RMPA )  of  20  tests: 

Ratio  of  means  and  variances  of  eigenvalues  of  the  jacknifed  submatrices 
(Ratio  -  observed  /  expected) 


Rep  r 

P 

Eigenvalue  1 
Mean  Var 

Eigenvalue  2 

Mean  Var 

Eigenvalue  3 

Mean  Var 

B 

0 

1.031 

0.933 

0.995 

0.884 

4.030 

6.007 

B 

0.0 

10 

1.031 

1.034 

1.574 

15.449 

9.730 

44.102 

B 

0.0 

25 

1.036 

1.074 

3.917 

28.726 

10.187 

52.047 

B 

0.0 

50 

1.056 

1.355 

19.748 

151.159 

10.350 

83.432 

B 

0.5 

10 

1.036 

0.920 

1.078 

1.779 

9.084 

80.253 

B 

0.5 

25 

1.042 

0.885 

2.646 

10.454 

9.763 

44.765 

B 

0.5 

50 

1.053 

0.909 

6.594 

5.886 

14.379 

133.295 

B 

0.7 

10 

1,036 

0.916 

1.045 

0.908 

7.086 

116.778 

B 

0.7 

25 

1.040 

0.934 

1.578 

3.836 

10.524 

62.910 

B 

0.7 

50 

1.046 

0.955 

3.114 

1.681 

14.278 

92.026 

R 

0 

1.001 

1.054 

0.928 

0.804 

1.790 

1.296 

R 

0.0 

10 

1.011 

1.067 

1.008 

0.776 

10.183 

370.061 

R 

0.0 

25 

1.012 

1.094 

2.600 

14.554 

11.918 

53.081 

R 

0.0 

50 

1.054 

1.746 

8.387 

31.775 

11.527 

78.033 

R 

0.5 

10 

1.019 

0.946 

1.000 

0.786 

6.685 

165.024 

R 

0.5 

25 

1.021 

0.888 

1.736 

4.773 

9.705 

36.368 

R 

0.5 

50 

1.030 

0.892 

3.554 

2.554 

14.892 

45.144 

R 

0.7 

10 

1.017 

0.944 

0.999 

0.770 

4.004 

44.388 

R 

0.7 

25 

1.017 

0.916 

1.199 

1.205 

10.082 

42.093 

R 

0.7 

50 

1.022 

0.966 

1.677 

0.763 

14.166 

68.059 

Notes: 

Ail  results  based  on  n=:80  items  and  N=2000  respondents. 
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Table  5: 

Revised  Modified  Parallel  Analysis  (RMPA)  of  20  tests: 

Correlations  of  eigenvalues  of  the  observed  and  the  expected  jacknifed  submatrices 


Rep  r 

Ev  1 

Ev  2 

Ev3 

Rep 

r  P 

Ev  1 

Ev  2 

Ev3 

B 

0 

0.996 

0.888 

0.650 

R 

0 

0.976 

0.956 

0.195 

B 

0.0 

10 

0.995 

-0.237 

0.603 

R 

0.0 

10 

0.995 

0.963 

-0.147 

B 

0.0 

25 

0.996 

-0.337 

0.655 

R 

0.0 

25 

0.996 

-0.414 

0.306 

B 

0.0 

50 

0.981 

-0.459 

-0.049 

R 

0.0 

50 

-0.641 

0.537 

0.321 

B 

0.5 

10 

0.997 

-0.256 

0.299 

R 

0.5 

10 

0.998 

0.968 

-0.129 

B 

0.5 

25 

0.998 

-0.320 

0.593 

R 

0.5 

25 

0.996 

-0.320 

0.394 

B 

0.5 

50 

0.990 

-0.180 

0.310 

R 

0.5 

50 

0.996 

-0.007 

0.218 

B 

0.7 

10 

0.997 

0.863 

-0.111 

R 

0.7 

10 

0.998 

0.968 

-0. 1 1 1 

B 

0.7 

25 

0.996 

-0.311 

0.608 

R 

0.7 

25 

0.995 

0.350 

0.140 

B 

0.7 

50 

0.997 

-0.206 

0.508 

R 

0.7 

50 

0.996 

0.060 

0.127 

Notes: 

All  results  based  on  n=80  items  and  N=2000  respondents. 
Ev  =  Eigenvalue 
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Table  6: 

Revised  Modified  Parallel  Analysis  (RMPA)  of  20  tests: 

*  Seven  detection  thresholds  based  on  the  distribution  of  the  standardized 
Weighted  Gaps  (SWGs)  based  on  the  ratio  of  the  first  observed  and  expected 
jacmifed  eigenvalues 


Threshold 

SWG  95  %  99  % 


Rep  r 

P 

Mean 

S.D. 

2.25 

Emp 

UChe 

Cheb 

Emp 

UChe 

Cheb 

B 

0 

0.94 

0.49 

2.25 

i.89 

2.42 

3.17 

2.38 

4.24 

5.88 

B 

0.0 

10 

0.91 

0.58 

2.25 

1.94 

2.67 

3.54 

2.58 

481 

6.76 

B 

0.0 

25 

0.97 

0.52 

.  2.25 

1.96 

2.54 

3.32 

2.87 

4.45 

6.19 

B 

0.0 

50 

0.84 

0.56 

2.25 

2.07 

2.53 

3.37 

3.09 

4.59 

6.46 

B 

0.5 

10 

0.90 

0.42 

2.25 

1.61 

2.15 

2.77 

1.92 

3.67 

5.06 

B 

0.5 

25 

0.96 

0.53 

2.25 

2.01 

2.56 

3.36 

2.93 

4.52 

6.30 

B 

0.5 

50 

0.97 

0.51 

2.25 

1.84 

2.50 

3.26 

2.26 

4.37 

6.07 

B 

0.7 

10 

0.98 

0.50 

2.25 

1.89 

2.48 

3.23 

2.80 

4.31 

5.97 

B 

0.7 

25 

0.95 

0.57 

2.25 

2.13 

2.65 

3.50 

2.66 

4.73 

661 

B 

0.7 

50 

0.89 

0.52 

2.25 

1.92 

2.45 

3.23 

2.13 

4.35 

6.08 

B 

Mean 

0.93 

0.52 

2.25 

1.93 

2.49 

3.27 

2.56 

4.40 

6.14 

R 

. 

0 

0.!>5 

0.44 

2.25 

1.65 

2.27 

2.93 

2.12 

3.89 

5.36 

R 

0,0 

10 

0.99 

0.56 

2.25 

1.85 

2.66 

3.50 

3.40 

4.72 

6.58 

R 

0.0 

25 

0.94 

0.54 

2.25 

1.90 

2.56 

3.37 

2.65 

4.54 

6.33 

R 

0.0 

50 

0.79 

0.51 

2.25 

1.77 

2.30 

3.06 

2.47 

4.16 

5.84 

R 

0,5 

10 

0.99 

0.48 

2.25 

1.82 

2.42 

3.14 

1.93 

4.18 

5.78 

R 

0.5 

25 

0.95 

0.52 

2.25 

2.03 

2.52 

3.31 

2.24 

4.44 

6.18 

R 

0.5 

50 

0.93 

0.53 

2.25 

1.82 

2.53 

3.33 

3.00 

4.48 

6.25 

R 

0.7 

10 

1.07 

0.61 

2.25 

2.16 

2.90 

3.81 

2.87 

5.12 

7.14 

R 

0.7 

25 

0.92 

0.45 

2.25 

1.95 

2.28 

2.95 

2.23 

3.93 

5.43 

R 

0.7 

50 

0.96 

0.47 

2.25 

1.85 

2.38 

3.09 

2.03 

4.12 

5.70 

R 

Mean 

0.95 

0.51 

2.25 

1.88 

2.48 

3.25 

2.49 

4.35 

6.06 

Mean 

0.94 

0.52 

2.25 

1.90 

2.49 

3.26 

2.53 

4.38 

6.10 

Notes: 

All  results  based  on  n=80  item’i  and  N=2000  respondents. 
Emp  =  Empirical  distribution 
UChe  =  Chebyshev  bound  assuming  unimodality 
Cheb  =  Chebyshev  bound 
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Table  7: 

Revised  Modified  Parallel  Analysis  (RMPA): 

Proportion  of  Standardized  Weighted  Gaps  (SWGs)  exceeding  each  of  the  seven  • 
thresholds  in  the  uncontaminated  unidimensional  test 


Eigenvalue 

2.250 

Emp 

Threshold 

95% 

UChe  Cheb 

Emp 

99% 

UChe 

Cheb 

1 

0.006 

0.051 

0.000 

0.000 

0.013 

0.000 

0.000 

2 

0.082 

0.177 

0.063 

0.019 

0,070 

0.006 

0.000 

3 

0.120 

0.215 

0.108 

0.038 

0.120 

0.006 

0.000 

Mean 

0.069 

0.148 

0.057 

0.019 

0.068 

0.004 

0.000 

Notes: 

All  results  based  on  n=80  items  and  N=2000  respondents. 
Emp  =  Empirical  distribution 
UChe  =  Chebyshev  bound  assuming  unimodality 
Cheb  =  Chebyshev  bound 


Table  8a: 

Revised  Modified  Parallel  Analysis  (RMPA)  of  20  tests: 

Maximal  Standardized  Weighted  Gap  (SWG)  and  significance  according  to  three 
types  of  thresholds  (First  eigenvalue) 


Rep 

r 

P 

Gap 

Max(SWG) 

2.25 

Significance*  No.  of  items 

95%  99%  Below  Above 

B 

0 

0.00007800 

2.37665 

1 

1 

40 

40 

B 

0.0 

10 

0.00011536 

2.58303 

1 

1 

29 

51 

B 

0.0 

25 

0.00017863 

2.87133 

1 

2 

52 

28 

B 

0.0 

50 

0.00074733 

3.09437 

1 

2 

46 

34 

B 

0.5 

10 

0.00007026 

1.92493 

0 

1 

42 

38 

B 

0.5 

25 

0.00023166 

2.93434 

1 

2 

20 

60 

B 

0.5 

50 

0.00011191 

2.25905 

1 

1 

49 

31 

B 

0.7 

10 

0.00011347 

2.80103 

1 

2 

35 

45 

B 

0.7 

25 

0.00059257 

2.66357 

1 

2 

76 

4 

B 

0.7 

50 

0.00035381 

2.12819 

0 

1 

76 

4 

R 

0 

0.00013665 

2.12450 

0 

1 

30 

50 

R 

0.0 

10 

0.00083803 

3.39892 

1 

2 

4 

76 

R 

0.0 

25 

0.00019422 

2.65083 

1 

2 

22 

58 

R 

0.0 

50 

0.00513373 

2.47295 

1 

2 

40 

40 

R 

0.5 

10 

0.00004417 

1.92817 

0 

1 

33 

47 

R 

0.5 

25 

0.00010552 

2.24198 

0 

1 

38 

42 

R 

0.5 

50 

0.00017510 

2.99795 

1 

2 

28 

52 

R 

0.7 

10 

0.00010866 

2.87002 

1 

1 

16 

64 

R 

0.7 

25 

0.00CO9185 

2.22529 

0 

1 

33 

47 

R 

0.7 

50 

0.00017596 

2.02883 

0 

1 

7 

73 

*Note: 

1  ->  Max(SWG)  >  Empiric^  percentile 

2  ">  Max(SWG)  >  Chebyshev  +  unimodality 

3  “>  Max(SWG)  >  Chebyshev 
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Table  8b: 

Revised  Modified  Parallel  Analysis  (RMPA)  of  20  tests: 

Maximal  Standardized  Weighted  Gap  (SWG)  and  significance  according  to  three 
types  of  thresholds  (Second  eigenvalue) 


Rep 

r 

P 

Gap 

Max(SWG) 

2.25 

Significance* 

95%  99% 

Nn.  of  items 

Below  Above 

B 

0 

0.002948 

4.6028 

1 

3 

2 

13 

67 

B 

0.0 

10 

0.154001 

10.3842 

1 

3 

3 

72 

8 

B 

0.0 

25 

0.058056 

5.5693 

1 

3 

2 

61 

19 

B 

0.0 

50 

0.420034 

2.9129 

1 

2 

0 

6 

74 

B 

0.5 

10 

0.061734 

7.9202 

1 

3 

3 

72 

8 

B 

0.5 

25 

0.020733 

3.9763 

1 

3 

1 

62 

18 

B 

0.5 

50 

0.016864 

2.7078 

1 

2 

1 

30 

50 

B 

0.7 

10 

0.017799 

3.6832 

1 

3 

1 

78 

2 

B 

0.7 

25 

0.017849 

4.2707 

1 

3 

1 

64 

16 

B 

0.7 

50 

0.061186 

2.6536 

1 

2 

I 

3 

77 

R 

« 

0 

0.002074 

2.6308 

1 

2 

1 

4 

76 

R 

0.0 

10 

0.001358 

3.1794 

1 

2 

0 

10 

70 

R 

0.0 

25 

0.041428 

5.1730 

1 

3 

2 

60 

20 

R 

0.0 

50 

0.063916 

7.3824 

1 

3 

3 

15 

65 

R 

0.5 

10 

0.002657 

2.7704 

1 

2 

1 

4 

76 

R 

0.5 

25 

0.032784 

6.7929 

1 

3 

3 

61 

19 

R 

0.5 

50 

0.027717 

3.0870 

1 

2 

1 

7 

73 

R 

0.7 

10 

0.000493 

2.7074 

1 

1 

0 

23 

57 

R 

0.7 

25 

0.005675 

2.7842 

1 

2 

I 

71 

9  - 

R 

0.7 

50 

0.005415 

2.7835 

1 

2 

1 

19 

61 

*Note: 

1  — >  Max(SWG)  >  Empirical  percentile 

2  “>  Max(SWG)  >  Chebyshev  +  unimodality 

3  ->  Max(SWG)  >  Chebyshev 


Table  8c: 

Revised  Modified  Parallel  Analysis  (RMPA)  of  20  tests: 

Maximal  Standardized  Weighted  Gap  (SWG)  and  significance  according  to  three 
types  of  thresholds  (Third  eigenvalue) 


Rep 

r 

P 

Gap 

Max(SWG) 

2.25 

Significance* 

95%  99% 

No.  of  items 

Below  Above 

B 

0 

0.197155 

5.0647 

1 

3 

2 

2 

78 

B 

0.0 

10 

0.407039 

3.5361 

1 

2 

1 

1 

79 

B 

0.0 

25 

0.087976 

4.3483 

1 

3 

1 

9 

71 

B 

0.0 

50 

0.065372 

3.5205 

1 

3 

1 

12 

68 

B 

0.5 

10 

0.110290 

3.4420 

1 

3 

1 

73 

7 

B 

0.5 

25 

0.138565 

4.4914 

1 

3 

1 

6 

74 

B 

0.5 

50 

0.237589 

5.509! 

1 

3 

2 

9 

71 

B 

0.7 

10 

0.184939 

7.1549 

1 

3 

3 

7! 

9 

B 

0.7 

25 

0.140215 

5.1461 

1 

3 

2 

8 

72 

B 

0.7 

50 

0.116452 

3.3573 

1 

3 

1 

7 

73 

R 

0 

0.013257 

3.6044 

1 

3 

1 

9 

71 

R 

0.0 

10 

0.419092 

9.4241 

1 

3 

3 

72 

8 

R 

0.0 

25 

0.115029 

4.1520 

1 

3 

1 

10 

70 

R 

0.0 

50 

0.225673 

8.3301 

1 

3 

3 

10 

70 

R 

0.5 

10 

0.287058 

10.3919 

1 

3 

3 

72 

8 

R 

0.5 

25 

0.118107 

5.3490 

1 

3 

2 

12 

68 

R 

0.5 

50 

0.464085 

4.3802 

1 

3 

1 

3 

77 

R 

0.7 

10 

0.111553 

6.4281 

1 

3 

2 

72 

8 

R 

0.7 

25 

0.072746 

3.5625 

1 

3 

1 

11 

69 

R 

0.7 

50 

0.272261 

4.7821 

1 

3 

2 

7 

73 

*Note: 

1  — >  Max(SWG)  >  Empirical  percentile 

2  -->  Max(SWG)  >  Chebyshev  +  unimodality 

3  ->  Max(SWG)  >  Chebyshev 
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Table  9: 

Revised  Modified  Parallel  Analysis  (RMPA)  of  20  tests: 

Maximal  Standardized  Weighted  Gaps  (SWGs)  and  significance  according  to  all 
eigenvalues 


Rep  r 

P 

Z1 

Max  (SWG) 

72 

Z3 

2.25-1- 

123 

95%* 

123 

99%* 

123 

B 

0 

2.38 

4.60 

5.06 

1  1  1 

1  33 

122 

B 

0.0 

10 

2.58 

10.38 

3.54 

1  1  1 

132 

1  3  1 

B 

0.0 

25 

2.87 

5.57 

4.35 

1  1  1 

23  3 

1  2  1 

B 

0.0 

50 

3.09 

2.91 

3.52 

1  1  1 

223 

1  0  1 

B 

0.5 

10 

1.92 

7.92 

3.44 

0  1  1 

133 

1  3  1 

B 

0.5 

25 

2.93 

3.98 

4.49 

1  1  1 

233 

1  1  1 

B 

0.5 

50 

2.26 

2.71 

5.51 

I  1  1 

123 

1  1  2 

B 

0.7 

10 

2.80 

3.68 

7.15 

1  1  1 

233 

1  1  3 

B 

0.7 

25 

2.66 

4.27 

5.15 

1  1  1 

233 

i  I  2 

B 

0.7 

50 

2.13 

2.65 

3.36 

0  1  1 

123 

1  1  1 

R 

« 

0 

2.12 

2.63 

3.60 

0  1  1 

123 

1  1  1 

R 

0.0 

10 

3.40 

3.18 

9.42 

1  1  1 

223 

103 

R 

0.0 

25 

2.65 

5.17 

4.15 

1  1  1 

233 

1  2  1 

R 

0.0 

50 

2.47 

7.38 

8.33 

1  1  1 

233 

133 

R 

0.5 

10 

1.93 

2.77 

10.39 

0  1  1 

123 

1  1  3 

R 

0.5 

25 

2.24 

6.79 

5.35 

0  1  1 

133 

132 

R 

0.5 

50 

3.00 

3.09 

4.38 

1  1  1 

223 

I  1  1 

R 

0.7 

10 

2.87 

2.71 

6.60 

1  1  1 

1  1  3 

102 

R 

0.7 

25 

2.23 

2.78 

3.56 

0  1  1 

123 

111  ^ 

R 

0.7 

50 

2.03 

2.78 

4.78 

0  1  1 

123 

1  1  2 

Note:  * 

1  ~>  Max(SWG)  >  Empirical  pen:entile 

2  — >  Max(SWG)  >  Chebyshev  +  unimodaiity 

3  — >  Max(SWG)  >  Chebyshev 


+  1  -  >  Max(SWG)  >  2.25 


Table  10: 


Revised  Modified  Parallel  Analysis  (RMPA)  of  10  short  tests: 

Total  number  of  items  eliminated  and  accuracy  of  the  elimination  procedure 


Rep 

r 

P 

Total 

Items  eliminated 

%  of  %  of 

"hits"  "false  alarms 

Significant 
’  Eigenvalue 

B 

0.0 

10 

8 

100 

0 

2 

B 

0.5 

10 

8 

100 

0 

2 

B 

0.7 

10 

9 

100 

1 

3 

R 

0.0 

10 

8 

100 

0 

3 

R 

0.5 

10 

8 

100 

0 

3 

R 

0.7 

10 

8 

100 

0 

3 

Mean 

8.2 

100 

0.2 

- 

B 

0.0 

25 

19 

95 

0 

2 

*B 

0.5 

25 

18 

90 

0 

2 

R 

0.0 

25 

20 

100 

0 

2 

R 

0.5 

25 

19 

95 

0 

2 

Mean 

19 

95 

0 

Mean 

- 

98 

0.1 

Note: 

Tests  shortened  by  99%  criterion 
*  These  tests  shortened  by  a  95%  criterion 


Table  11: 

Modified  Parallel  Analysis  (MPA)  of  10  short  tests: 

The  first  three  eigenvalues  for  the  observed  and  expected  matrices,  and  their  ratios 


Eigenvalue  1  Eigenvalue  2  Eigenvalue  3 


Rep  r 

P 

Exp 

Obs 

Obs/Exp 

Exp 

Obs 

Obs/Exp 

Exp 

Obs 

Obs/Exp 

B 

0.0 

10 

21.86 

22.71 

1.04 

1.62 

1.65 

1.02 

0.17 

0.62 

3.64 

B 

0.0 

25 

17.89 

18.78 

1.05 

1.42 

1.48 

1.05 

0.14 

0.56 

3.86 

B 

0.5 

10 

21.95 

22.71 

1.03 

1.65 

1.65 

1.00 

0.18 

0.62 

3.51 

B 

0.5 

25 

18.00 

18.85 

1.05 

1.40 

1.49 

1.06 

0.15 

0.58 

3.89 

B 

0.7 

10 

21.49 

22.27 

1.04 

1.58 

1.61 

1.01 

0.17 

0.62 

3.63 

R 

0.0 

10 

23.35 

23.84 

1.02 

2.84 

2.85 

1.01 

0.25 

0.54 

2.19 

R 

0.0 

25 

19.17 

19.79 

1.03 

2.37 

2.30 

0.98 

0.19 

0.50 

2.63 

R 

0.5 

10 

22.79 

23.21 

1.02 

2.70 

2.71 

1.00 

0.28 

0.58 

2.10 

R 

0.5 

25 

19.27 

19.73 

1.02 

2.33 

2.45 

1.05 

0.24 

0.53 

2.21 

R 

0.7 

10 

22.83 

23.21 

1.02 

2.73 

2.71 

0.99 

0.28 

0.58 

2.04 

Notes: 

All  results  based  on  N=2000  respondents,  and  various  number  of  items. 
Exp  =  Derived  from  matrix  of  expected  correlations 
Obs  s  Derived  from  matrix  of  observed  correlations. 


Table  12: 


Revised  Modified  Parallel  Analysis  (RMPA)  of  10  short  tests: 

Ratio  of  means  and  variances  of  eigenvalues  of  the  jacknifed  submatrices 
(Ratio  -  observed  /  expected) 


Rep 

r 

P 

Eigenvalue  1 

Mean  Var 

Eigenvalue  2 

Mean  Var 

Eigenvalue  3 

Mean  Var 

B 

0.0 

10 

1.039 

0.928 

1.023 

1.066 

3.660 

5.473 

B 

0.0 

25 

1.049 

0.923 

1.046 

1.032 

3.889 

B 

0.5 

10 

1.035 

0.922 

1.005 

0.983 

3.526 

5.112 

B 

0.5 

25 

1.047 

0.959 

1.064 

1.065 

3.913 

6.538 

B 

0.7 

10 

1.036 

0.919 

1.015 

1.072 

3.641 

5.785 

R 

0.0 

10 

1.021 

0.975 

1.005 

0.816 

2.202 

1.152 

R 

0.0 

25 

1.033 

0.947 

0.976 

0.760 

2.648 

1.670 

R 

0.5 

10 

I.0I8 

0.960 

1.001 

0.785 

2.107 

1.299 

R 

0.5 

25 

1.024 

0.997 

1.054 

0.863 

2.227 

1.418 

R 

0.7 

10 

1.017 

0.949 

0.991 

0.772 

2.047 

-47- 


Table  13a: 

Revised  Modified  Parallel  Analysis  (RMPA)  of  10  short  tests: 

Maximal  Standardized  Weighted  Gap  (SWG)  and  significance  according  to 
three  types  of  thresholds 
First  Eigenvalue 


Rep 

r 

P 

Gap 

Max(SWG) 

2.25 

Signiflcance*  No.  of  items 

95%  99%  Below  Above  Total 

B 

0.0 

10 

0.00012884 

4.38366 

1 

1 

1  21 

51 

72 

B 

0.0 

25 

0.00017634 

4.77832 

1 

2 

1  33 

28 

61 

B 

0.5 

10 

0.00011606 

4.63637 

1 

1 

1  45 

27 

72 

B 

0.5 

25 

0.00017343 

1.01824 

0 

1 

I  10 

52 

62 

B 

0.7 

10 

0.00008747 

2.41954 

1 

1 

1  26 

45 

71 

R 

0.0 

10 

0.00005516 

3.73614 

1 

1 

1  34 

38 

72 

R 

0.0 

25 

0.00025199 

0.44202 

0 

1 

1  8 

52 

60 

R 

0.5 

10 

0.00006304 

3.83447 

1 

1 

1  32 

40 

72 

R 

0.5 

25 

0.00010094 

2.84225 

1 

1 

1  39 

22 

61 

R 

0.7 

10 

0.00006237 

0.75571 

0 

1 

1  32 

40 

72 

*Note: 

1  ->  Max(SWG)  >  Empirical  percentile 

2  “>  Max(SWG)  >  Chebyshev  +  unimodality 
3->  Max(SWG)  >  Chebyshev 
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Table  13b: 

Revised  Modified  Parallel  Analysis  (RMPA)  of  1 0  short  tests: 

Maximal  Standardized  Weighted  Gap  (SWG)  and  significance  according  to  three 
types  of  thresholds. 

Second  Eigenvalue 


Rep 

r 

P 

Gap 

Max(SWG) 

B 

0.0 

10 

0.0012185 

6.23878 

B 

0.0 

25 

0.0025284 

6.06745 

B 

0.5 

10 

0.0062079 

5.52540 

B 

0.5 

25 

0.0241364 

5.76306 

B 

0.7 

10 

0.0053278 

6.40449 

R 

0.0 

10 

0.0007408 

0.16707 

R 

0.0 

25 

0.0023976 

1.68404 

R 

0.5 

10 

0.0059670 

4.56087 

R 

0.5 

25 

0.0039343 

3.44740 

R 

0.7 

10 

0.0012628 

4.00428 

Significance*  No.  of  items 
95%  99%  Below  Above  Total 


0 

0 

1 

1 

1 


3 

2 

2 

3 

3 

0 

3 

2 

2 

3 


1 

1 

1 

3 

2 

0 

3 

1 

1 

3 


20 

10 

69 

1 

66 

21 

5 

2 

4 

9 


52 

51 

3 

61 

5 

51 

55 

70 

57 

63 


72 

61 

72 

62 

71 

72 
60 
72 
61 
72 


*Note: 

1  ->  Max(SWG)  >  Empirical  percentile 

2  ->  Max(SWG)  >  Chcbyshev  +  unimodality 

3  ->  Max(SWG)  >  Chcbyshev 
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Table  13c: 

Revised  Modified  Parallel  Analysis  (RMPA)  of  10  short  tests: 

Maximal  Standardized  Weighted  Gap  (SWG)  and  significance  according  to  three 
types  of  thresholds 
Third  Eigenvalue 


Rep 

r 

P 

Gap 

Max(SWG) 

2.25 

Significance* 

95%  99% 

No.  of  items 

Below  Above  Total 

B 

0.0 

10 

0.056692 

4.94463 

1 

2 

1 

4 

68 

72 

B 

0.0 

25 

0.170522 

5.36876 

1 

2 

1 

3 

58 

61 

B 

0.5 

10 

0.015446 

4.50512 

1 

1 

0 

14 

58 

72 

B 

0.5 

25 

0.240886 

4.33449 

1 

3 

3 

1 

61 

62 

B 

0.7 

10 

0.017104 

3.98031 

1 

3 

1 

14 

57 

71 

R 

0.0 

10 

0.045500 

4.07539 

1 

1 

1 

5 

67 

72 

R 

0.0 

25 

0.045556 

4.54326 

1 

3 

3 

10 

50 

60 

R 

0.5 

10 

0.015318 

1.00964 

0 

0 

0 

14 

58 

72 

R 

0.5 

25 

0.021710 

0.80738 

0 

0 

0 

14 

47 

61 

R 

0.7 

10 

0.022073 

1.57497 

0 

3 

2 

15 

57 

72 

*Note: 

1  ->  Max(SWG)  >  Empirical  percentile 

2  “>  Max(SWG)  >  Chebyshev  +  unimodality 

3  “>  Max(SWG)  >  Chebyshev 
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FIGURE  CAPTIONS 

Figure  I :  Distribution  of  SWGs  based  on  the  ratio  of  the  second  pair  of  eigenvaJues 
(Rep=B,  r=0.0,  p=10). 

Figure  2;  Distribution  of  SWGs  based  on  the  ratio  of  the  second  pair  of  eigenvalues 
(Rep=B.  r=0.5,  p=I0). 

Figure  3:  Distribution  of  SWGs  based  on  the  ratio  of  the  third  pair  of  eigenvalues 
(Rep=B,  r=0.7,  p=10). 

Figure  4;  Distribution  of  SWGs  based  on  the  ratio  of  the  third  pair  of  eigenvalues 
(Rep=R.  r=0.0,  p=10). 

Figure  5:  Distribution  of  SWGs  based  on  the  ratio  of  the  third  pair  of  eigenvalues 
(Rep=R,  r=0.5.  p=10). 

Figure  6:  Distribution  of  SWGs  based  on  the  ratio  of  the  third  pair  of  eigenvalues 
(Rep=R.  r=0.7,  p=10). 

Figure  7:  Distribution  of  SWGs  based  on  the  ratio  of  the  second  pair  of  eigenvalues 
(Rep=B.  r=0.0,  p=25). 

Figure  8:  Distribution  of  SWGs  based  on  the  ratio  of  the  second  pair  of  eigenvalues 
(Rep5=B.  r=0.5.  p=25). 


Figure  9:  Distribution  of  SWGs  based  on  the  ratio  of  the  second  pair  of  eigenvalues 
(RepsR,  n=0,  p=25). 

Figure  10;  Distribution  of  SWGs  based  on  the  ratio  of  the  second  pair  of  eigenvalues 
(Rep=R,  r=0.5,  p=25). 


Figure  1 1 :  RCXH  curves  for  the  ratio  of  the  first  and  second  pair  of  eigenvalues 
(Rep=B,  r=0,0,  p=10). 

Figure  12:  ROC  curves  for  the  ratio  of  the  first  and  second  pair  of  eigenvalues 
(Rep=B,  r=0.5,  p=10). 

Figure  13:  ROC  curves  for  the  ratio  of  the  first  and  third  pair  of  eigenvalues 
(Rep=B,  r=0.7.  p=10). 

Figure  14;  ROC  curves  for  the  ratio  of  the  first  and  third  pair  of  eigenvalues 
(Rep=R,  p=0.0,  p=10). 

Figure  15:  ROC  curves  for  the  ratio  of  the  first  and  third  pair  of  eigenvalues 
(Rep=R,  r=:0.5,  p=10). 

Figure  16:  ROC  curves  for  the  ratio  of  the  first  and  third  pair  of  eigenvalues 
(Rep=R,  r=0.7.  p=10). 

Figure  17:  ROC  curves  for  the  ratio  of  the  first  and  second  pair  of  eigenvalues 
(Rep=B,  r=0.0,  p=25). 
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Figure  18:  ROC  curves  for  the  ratio  of  the  first  and  second  pair  of  eigenvalues 
(Rep=B,  r=0.5,  p=25). 

Figure  19:  ROC  curves  for  the  ratio  of  the  first  and  second  pair  of  eigenvalues 
(Rep=:R,  r=0,  p=25). 

Figure  20:  ROC  curves  for  the  ratio  of  the  first  and  second  pair  of  eigenvalues 
(Rep=R,  r=0.5,  p=25). 


Figure  3:  Distribution  of  SWGs  batsd  on  ths  ratios  of  tha 
third  pair  of  aiganvaluas 
(REPbB  RaO.7  PsIO) 
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Figure  4:  Distribution  of  SWGs  batod  on  tfia  ratios  of  tha 
third  pair  of  aiganvaiuas 
iREPsB  RsO  PsIOl 


RL, 

SWG, 

, 

8.08 

0.00 

2 

8.11 

0.94 

3 

8.34 

3.70 

4 

8.75 

5.84 

5 

9.11 

6.36 

6 

9.21 

3.81 

7 

9.55 

7.40 

8 

9.72 

5.78 

9 

10.14 

9.42 

10 

10.17 

2.43 

11 

10.18 

1.49 

12 

10.18 

0.36 

13 

10.19 

1.78 

14 

10.19 

0.45 

15 

10.19 

0.49 

16 

10.19 

1.07 

17 

10.19 

0.35 

18 

10.19 

0.56 

19 

10.20 

0.60 

20 

10.20 

0.92 

21 

10.20 

0.32 

22 

10.20 

0.33 

23 

10  20 

0  61 

24 

10.20 

0.59 

25 

10.20 

0.44 

26 

10.20 

1.04 

27 

to  20 

0.88 

28 

10.20 

0.65 

29 

10.21 

1.17 

30 

10.21 

0.51 

31 

10.21 

0.59 

32 

10.21 

1.17 

33 

10.21 

0.43 

34 

10.21 

0.51 

35 

10.21 

0.53 

36 

10.21 

0.43 

37 

10.21 

0.00 

38 

10.21 

1.20 

39 

10.22 

1.32 

40 

10.22 

0.94 

41 

10.22 

III 

42  10.22  1.49 

43  10.23  1.32 

44  10.23  0.27 

45  10.23  0.46 

46  10.23  0.55 

47  1  0.23  1.44 

48  10.24  1.86 

49  10.24  0.87 

50  10.24  1.58 

51  10.25  1.20 

52  10.25  1.53 


53 

10.25 

1.13 

54 

10.25 

0.69 

55 

10.26 

1.67 

56 

10.27 

2.57 

57 

10.27 

0.44 

58 

10.29 

3.01 

59 

10.30 

2.14 

60 

10.30 

0.19 

61 

10.31 

2.41 

62 

10.33 

2.79 

63 

10.34 

1.94 

64 

10.37 

3.23 

65 

10.37 

1.23 

66 

10.46 

5  69 

67 

10.49 

3.05 

68 

10.50 

1.68 

69 

10.52 

2.59 

70 

10.53 

1.62 

71 

10.54 

1.80 

72 

10.56 

2.06 

73 

10  63 

3.94 

74 

10.66 

2.01 

75 

10.77 

4.28 

76 

10.87 

3.68 

77 

10.87 

0.92 

78 

10.92 

1.97 

79 

10.99 

2.02 

80 

1 1.97 

5.34 

SWG- 


Figure  7:  Dwtribution  of  SWGs  ba««d  on  tho  ratios  of  tha 
sacond  pair  of  alganvaluat 
(REPsB  RsO  Ps25) 
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16 
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17 
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1.07 

18 

3.82 

1.99 

19 

3.83 
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20 

3.89 
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21 
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4.19 
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Figure  10:  DI*trMiiitlon  of  SWQ«  based  on  tha  ratios  of  tha 
second  pair  of  alpanvaluas 
(REPbR  RsO.5  Pa26) 
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Figure  1 1 :  ROC  curves  for  the  ratio  of  the  first  and  second  pairs  of  eiganvaiues 
REPsB  Rs0.0  PsIO 
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Figure  12:  ROC  curv*s  for  th*  ratio  of  th«  Rrat  and  sacond  pairs  of  •iganvalues 
REPsB  RsO.5  PsIO 
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Figure  13:  ROC  curvas  for  the  ratio  of  the  first  and  third  pairs  of  eigenvalues 
REPsB  RsO.7  PsIO 
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Figure  1 S:  ROC  curves  for  the  ratio  of  the  first  and  third  pairs  of  eigenvalues 
REPsR  RsO.5  PsIO 
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Figure  16;  ROC  curves  for  th*  ratio  of  th«  first  and  third  pairs  of  'sigenvalues 
REPsR  RrO.7  PsIO 


Figure  17:  ROC  curv«s  for  the  ratio  of  the  first  and  second  pairs  of  eigenvalues 
REPsB  RsO.O  Ps2S 
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Figure  18:  ROC  curvas  for  the  ratio  of  the  first  and  second  pairs  of  eigenvalues 
REP=B  RsO.5  P=25 
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Figure  ]9:  ROC  curvas  for  the  ratio  of  the  first  and  second  pairs  of  eigenvalues 
REPsR  Rs0.0  P=25 
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Figure  20:  ROC  curvas  fof  th*  ratio  of  th*  flrat  and  «*cond  pair*  of  eigenvalu** 
REPrR  RsO.5  PsaS 
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APPENDIX  I: 

Parameters  of  the  items  used  in 
the  R  tests 


Item 

a 

b 

C 

001 

0.70 

-1.97 

0.00 

002 

1.01 

-1.26 

0.08 

003 

1.19 

-1  15 

0.02 

004 

0.81 

-0.64 

0.28 

005 

0.57 

-1.41 

0.03 

006 

0.91 

-1.07 

0.00 

007 

0.60 

-1.50 

0.06 

008 

0.54 

-1.08 

0.20 

009 

1.25 

-2.06 

0.03 

010 

1.30 

-1.45 

0.00 

on 

1.13 

-1.21 

0.02 

012 

1.21 

•0.89 

0.09 

013 

1.01 

-067 

0.11 

014 

1.82 

4)54 

0.16 

OIS 

1.36 

-0.67 

014 

016 

1.57 

-0.06 

0.34 

017 

2.02 

-0.79 

0.23 

018 

0.72 

-0.74 

0.16 

019 

1.06 

-0.63 

0.09 

020 

1  50 

0.08 

0.22 

021 

1.57 

0.93 

0.19 

022 

1.23 

1.19 

0.19 

023 

0.86 

0.53 

027 

024 

081 

0.65 

0.27 

025 

1.38 

0.11 

0.20 

026 

0.72 

0.22 

0.27 

027 

1.70 

0.34 

0.22 

028 

I.SS 

0.48 

0.27 

029 

1.35 

046 

0.30 

030 

1.79 

0.98 

0.19 

031 

1.62 

0.41 

0.17 

032 

2.44 

0.84 

0.19 

033 

2.44 

1.02 

017 

034 

2.44 

1.61 

0.31 

035 

2.44 

0.87 

0.23 

036 

1.40 

0,53 

0.20 

037 

0.81 

0.41 

0.20 

038 

1.43 

0.79 

020 

039 

1.23 

1.34 

0.23 

040 

0.79 

1.39 

0.17 

041 

0.57 

-1.87 

0.03 

042 

0.80 

-166 

0.02 

043 

1.07 

-1.38 

0.00 

044 

0.85 

-106 

0.00 

045 

1.32 

-0.17 

0.28 

046 

I.IO 

-1,07 

0.08 

047 

0.71 

-0.67 

0.08 

048 

1.31 

-1.17 

0.02 

049 

173 

-1.22 

0.03 

050 

1.60 

-1.29 

0.03 

051 

1.03 

-1  13 

0.03 

052 

1.21 

•0,90 

0.00 

053 

0.63 

-0.02 

0.09 

054 

1.14 

-0.19 

0.19 

055 

0.85 

-0.83 

0.08 

056 

1.14 

0.20 

0.16 

057 

0.71 

0.86 

019 

058 

1.36 

-0.81 

0.06 

059 

0.95 

-0,27 

0.09 

060 

1.29 

0.43 

0,31 

061 

0.85 

0.28 

013 

062 

1.89 

0.88 

0.17 

063 

0.73 

077 

017 

064 

0.97 

0,56 

0.30 

065 

1.41 

0.45 

0.16 

066 

1.82 

0.66 

030 

067 

1.70 

-0.06 

019 

068 

1.70 

035 

0.28 

069 

1.48 

099 

0.16 

070 

1.72 

093 

0,19 

071 

1.80 

0.75 

016 

072 

2.47 

1.09 

022 

073 

1  56 

1.12 

031 

074 

2.04 

1.20 

0.33 

075 

199 

1.29 

0.22 

076 

1.09 

0.80 

027 

077 

1  10 

0.61 

0.17 

078 

1  82 

1.19 

022 

079 

2.47 

1  24 

030 

080 

2.04 

1.61 

0.17 
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APPENDIX  2: 

Parameters  of  the  items 
used  in  the  B  tests 


Item  a 


b  c 


2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 
17 
IS 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 

42 

43 

44 

45 

46 

47 

48 

49 

50 

51 

52 

53 

54 

55 

56 

57 

58 

59 

60 
61 
62 

63 

64 

65 

66 

67 

68 

69 

70 

71 

72 

73 

74 

75 

76 

77 

78 

79 

80 


1.41 
1.00 

1.19 
135 
1.35 
153 
1.45 
1.09 
0.93 
0.89 

1.25 
1.37 
0.88 

1.30 
0.81 
1.02 
0.96 
0.99 

1.42 
0.62 

1.19 
I  14 
0.96 
1.23 
0.89 
1.12 
0.65 
0.83 
0.90 

1.27 
0.90 
1.32 

1.42 
145 

1.30 
1.17 
1.39 
0.78 
1.48 

1.31 

1.54 

1.54 
1.13 

1.27 
0.84 
1.21 
1.06 

1.19 
0.85 
0.65 

1.32 
1.30 
0.79 
107 
131 
0.93 
1.08 

1.27 

1.26 
1.06 
0.93 
124 
120 
1.40 
0.93 
1.08 
075 

1.26 
1.04 
0.74 
1.44 
089 
0.73 
095 
0.77 
086 
I  29 

1.55 
I  35 
I  26 


0.42 

0.25 

0.06 

0.28 

0.40 

0.20 

1.61 

0.27 

-0.30 

0.25 

I.IS 

0.14 

0.60 

0.10 

0.68 

0.22 

0.13 

0.28 

-I.OI 

0.20 

-0.51 

0.23 

-0.90 

OJO 

-040 

022 

-0.21 

0.20 

0.77 

0.22 

-0.69 

0.28 

0.81 

0.11 

0.48 

0.28 

084 

0.12 

0.36 

0.20 

1.17 

0.21 

0.66 

0.10 

-0.02 

0.18 

-0.19 

0.19 

-G.20 

0.13 

-0.79 

0.20 

0.72 

0.26 

-0.29 

0.17 

•1.14 

0.24 

-085 

0.28 

-0.25 

0.24 

■1.76 

024 

•0.28 

0.(8 

1.44 

0.10 

1.48 

0.14 

-0.1 1 

0.29 

1.99 

0.20 

1.73 

0.14 

-0.10 

0.23 

0.72 

012 

0.10 

0.15 

1.50 

017 

0.10 

0.24 

-0.39 

0.10 

-111 

0.29 

-0.12 

0.25 

053 

0.28 

-1  32 

0.22 

-0.42 

0.14 

0.81 

0.23 

1.30 

0.27 

1.20 

014 

-1  49 

0.24 

-l.% 

0.17 

047 

0.10 

1.25 

0.19 

0.69 

0.28 

•0.18 

0.14 

1.02 

0.20 

0.41 

0.20 

0.10 

0.27 

-0.10 

016 

-0.21 

0.12 

0.19 

0.14 

0,92 

0.12 

1.60 

017 

0.30  0'4 

-0.13  0.19 

-0.26  0.27 

1.38  0.20 

-I.SI  026 

0.38  0>8 

0.85  0  28 

0  25  0  24 

016  0  22 

1  70  022 

0,74  029 

014  019 

.039  020 

080  0,17 
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APPENDIX  3: 

Mean  values  (and  standard  deviations)  of  the  estimates  of 
parameter  a  for  items  loaded  on  the  dominant  (main)  and  the 
nuisance  (cont.)  dimension 


P  = 

main 

(n=72) 

10 

cont. 

(n=8) 

P  = 

main 

(n=60) 

25 

cont. 

(n=20) 

p  =  50 

main 

(n=40) 

! 

cont. 

(n=40) 

Rep  =  R 

True 

1.31 

1.45 

1.31 

1.38 

1.29 

1.36 

(.51) 

(.56) 

(.50) 

(.55) 

(.48) 

(.54) 

p=0 

1.24 

1.37 

1.23 

1.30 

1.20 

1.29 

(.47) 

(.47) 

(.45) 

(.55) 

(.42) 

(.52) 

r=.7 

1.24 

.70 

1.20 

.80 

.97 

1.06 

(.50) 

(.15 

(.45) 

(.22) 

(.29) 

(.38) 

r=.5 

1.23 

.53 

1.21 

.60 

.86 

.96 

(.48) 

(.12) 

(.45) 

(.14) 

(.23) 

(.34) 

r=.0 

1.25 

.57 

1.25 

.74 

.27 

1.17 

(.49) 

(.77) 

(.51) 

(.88) 

(.01) 

(.47) 

Rep  =  B 

True 

1.13 

1.05 

1.12 

1.12 

1.13 

1.12 

(.25) 

(.21) 

(.25) 

(.24) 

(.25) 

(.24) 

pssO 

1.03 

.92 

1.02 

1.02 

1.05 

.99 

(.26) 

(.16) 

(.26) 

(.25) 

(.28) 

(.23) 

T=J 

1.03 

.57 

.99 

.58 

.82 

.83 

(.25) 

(.11) 

(.23) 

(.11) 

(.21) 

(.17) 

r=.5 

1.03 

.44 

1.00 

.51 

.83 

.67 

(.25) 

(.06) 

(.25) 

(.10) 

(.16) 

(.14) 

r=.0 

1.02 

.28 

1.00 

.93 

.92 

.27 

(.28) 

(.0) 

(.26) 

(.99) 

(.19) 

(.02) 
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APPENDIX  4: 

Mean  values  (and  standard  deviations)  of  item  reliabilities 
for  items  loaded  on  the  dominant  (main)  and  the  nuisance 
(cont.)  dimension 


P  = 

main 

(n=72) 

10 

cont. 

(n=8) 

P  = 

main 

(n=60) 

25 

cont. 

(n=20) 

p  =  50 

main 

(n=40) 

cont. 

(n=40) 

Rep  =  R 

True 

.37 

.44 

.37 

.36 

.39 

.35 

(.13) 

(.15) 

(.14) 

(.12) 

(.13) 

(.14) 

p=0 

.34 

.42 

.36 

.35 

.37 

.34 

(.14) 

(.16) 

(.15) 

(.13) 

(.14) 

(.15) 

T=J 

.36 

.22 

.36 

.21 

.33 

.28 

(.14) 

(.11) 

(.15) 

(.08) 

(.13) 

(.11) 

r=.5 

.36 

.12 

.36 

.12 

.29 

.24 

(.14) 

(.06) 

(.15) 

(.04) 

(.12) 

(.10) 

t^.O 

.37 

.03 

.36 

.02 

.04 

.30 

(.14) 

(.02) 

(.15) 

(.01) 

(.01) 

(.13) 

Rep  =  B 

True 

.34 

.37 

.35 

.31 

.35 

.33 

(.12) 

(.06) 

(.11) 

(.13) 

(.12) 

(.11) 

p=0 

.32 

.35 

.33 

.30 

.34 

.31 

(.12) 

(.05) 

(.11) 

(.14) 

(.12) 

(.12) 

r=.7 

.33 

.18 

.32 

.17 

.27 

.26 

(.12) 

(.05) 

(.12) 

(.09) 

(.11) 

(.10) 

r=.5 

.33 

.10 

.32 

.11 

.22 

.25 

(.12) 

(.01) 

(.12) 

(.04) 

(.10) 

(.08) 

r=.0 

.33 

.02 

.32 

.01 

.04 

.28 

(.12) 

(.01) 

(.12) 

(.01) 

(.02) 

(.11) 
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APPENDIX  5a: 

Revised  Modified  Parallel  Analysis  (RMPA)  of  18  tests: 
Proportion  of  Standardized  Weighted  Gaps  (SWGs) 
above  the  2.25  detection  threshold 


Finjt  EV. 

10 

P 

25 

50 

Mean 

R 

0 

0.025 

0.025 

0.019 

0.023 

5 

0.000 

0.013 

0.013 

0.009 

7 

0.025 

0.006 

0.000 

0.010 

Mean 

0.017 

0.015 

0.011 

0.014 

Second  EV. 

10 

P 

25 

50 

Mean 

R 

0 

0.101 

0.127 

0.158 

0.129 

5 

0.057 

0.120 

0.051 

0.076 

7 

0.063 

0.082 

0.038 

0.061 

Mean 

0.074 

0.110 

0.082 

0.089 

Third  EV. 

10 

P 

25 

50 

Mean 

R 

0 

0.209 

0.146 

0.241 

0.199 

5 

0.171 

0.133 

0.209 

0.171 

7 

0.184 

0.139 

0.152 

0.158 

Mean 

0.188 

0.139 

0.201 

0.176 
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APPENDIX  5b: 

Revised  Modified  Parallel  Analysis  (RMPA)  of  18  tests: 
Proportion  of  Standardized  Weighted  Gaps  (SWGs) 
above  95th  empirical  percentile 


First  EV. 

10 

P 

25 

50 

Mean 

R 

0 

0.051 

0.051 

0.051 

0.051 

5 

0.051 

0.051 

0.051 

0.051 

7 

0.051 

0.051 

0.051 

0.051 

Mean 

0.051 

0.051 

0.051 

0.051 

Second  EV. 

10 

P 

25 

50 

Mean 

R 

0 

0.146 

0.241 

0.184 

0.190 

5 

0.139 

0.184 

0.114 

0.146 

7 

0.089 

0.133 

0.082 

0.101 

Mean 

0.125 

0.186 

0.127 

0.146 

Third  EV. 

10 

P 

25 

50 

Mean 

R 

0 

0.335 

0.222 

0.184 

0.247 

5 

0.348 

0.158 

0.228 

0.245 

7 

0.228 

0.184 

0.234 

0.215 

Mean 

0.304 

0.188 

0.215 

0.236 

APPENDIX  Sc: 


-  A7- 


Revised  Modified  Parallel  Analysis  (RMPA)  of  18  tests: 
Proportion  of  Standardized  Weighted  Gaps  (SWGs) 
above  95th  percentile  (Chebyshev  inequality  +  unimodality) 


First  EV. 

10 

P 

25 

50 

Mean 

R 

0 

0.006 

0.019 

0.019 

0.015 

5 

0.000 

0.013 

0.006 

0.006 

7 

0.006 

0.006 

0.000 

0.004 

Mean 

0.004 

0.013 

0.008 

0.008 

Second  EV. 

10 

P 

25 

50 

Mean 

R 

0 

0.057 

0.076 

0.152 

0.095 

5 

0.057 

0.095 

0.032 

0.061 

7 

0.025 

0.044 

0.019 

0.029 

Mean 

0.046 

0.072 

0.068 

0.062 

Third  EV. 

10 

P 

25 

50 

Mean 

R 

0 

0.171 

0.089 

0.209 

0.156 

5 

0.184 

0.095 

0.1 14 

0.131 

7 

0.127 

0.114 

0.120 

0.120 

Mean 


0.161 


0.099 


0.148 


0.136 


-  A8- 


APPENDIX  Sd; 

Revised  Modified  Parallel  Analysis  (RMPA)  of  18  tests: 
Proportion  of  Standardized  Weighted  Gaps  (SWGs) 
above  95th  percentile  ( Chebyshev  inequality) 


First  EV. 


10 

25 

50 

Mean 

R 

0 

0.000 

0.000 

0.000 

0.000 

5 

0.000 

0.000 

0.000 

0.000 

7 

0.000 

0.000 

0.000 

0.000 

Mean 

0.000 

0.000 

0.000 

0.000 

Second  EV. 

10 

P 

25 

50 

Mean 

R 

0 

0.006 

0.044 

0.101 

0.050 

5 

0.013 

0.032 

0.000 

0.015 

7 

0.006 

0.006 

0.000 

0.004 

Mean 

0.008 

0.027 

0.034 

0.023 

Third  EV. 


-  A9- 


APPENDIX  5e: 

Revised  Modified  Parallel  Analysis  (RMPA)  of  18  tests: 
Proportion  of  Standardized  Weighted  Gaps  (SWGs) 
above  99th  empirical  percentile 


First  £V. 

to 

P 

25 

50 

Mean 

R 

0 

0.013 

0.013 

0.013 

0.013 

5 

0.013 

0.013 

0.013 

0.013 

7 

0.013 

0.013 

0.013 

0.013 

Mean 

0.013 

0.013 

0.013 

0.013 

Second  EV. 

10 

P 

25 

50 

Mean 

R 

0 

0.044 

0.063 

0.127 

0.078 

5 

0.108 

0.070 

0.044 

0.074 

7 

0.013 

0.044 

0.057 

0.038 

Mean 

0.055 

0.059 

0.076 

0.063 

Third  EV. 

10 

P 

25 

50 

Mean 

R 

0 

0.146 

0.076 

0.1.39 

0. 1 20 

5 

0.272 

0.108 

0.127 

0.169 

7 

0.114 

0.114 

0.171 

0.133 

Mean 

0.177 

0.099 

0.146 

0.141 

APPENDIX  5f: 


-  AIO- 


Revised  Modified  Parallel  Analysis  (RMPA)  of  18  tests: 
Proportion  of  Standardized  Weighted  Gaps  (SWGs) 
above  99th  percentile  (Chebyshev  inequality  +  unimodality) 


First  EV. 

10 

P 

25 

50 

Mean 

R 

0 

0.000 

0.000 

0.000 

0.000 

5 

0.000 

0.000 

0.000 

0.000 

7 

0.000 

0.000 

0.000 

0.000 

Mean 

0.000 

0.000 

0.000 

0.000 

Second  EV. 


10 

25 

50 

Mean 

R 

0 

0.006 

0.013 

0.044 

0.021 

5 

0.006 

0.006 

0.000 

0.004 

7 

0.000 

0.000 

0.000 

0.000 

Mean 

1 

0.004 

0.006 

0.015 

0.008 

Third  EV. 


10 

25 

50 

Mean 

R 

0 

0.044 

0.000 

0.057 

0.034 

5 

0.051 

0.013 

0.006 

0.023 

7 

0.057 

0.006 

0.006 

0.023 

Mean 

0.051 

0.006 

0.023 

0.027 

-  All  - 


APPENDIX  5g: 

Revised  Modified  Parallel  Analysis  (RMPA)  of  18  tests: 
Proportion  of  Standardized  Weighted  Gaps  (SWGs) 
above  99th  percentile  (Chebyshev  inequality) 


First  EV. 

10 

P 

25 

50 

Mean 

R 

0 

0.000 

0.000 

0.000 

0.000 

5 

0.000 

0.000 

0.000 

0.000 

7 

0.000 

0.000 

0.000 

0.000 

Mean 

0.000 

0.000 

0.000 

0.000 

10 

25 

50 

Mean 

R 

0 

0.006 

0.000 

0.013 

0.006 

5 

0.006 

0.006 

0.000 

0.004 

7 

0.000 

0.000 

0.000 

0.000 

Mean 

0.004 

0.002 

0.004 

0.003 

Third  EV. 

10 

P 

25 

50 

Mean 

R 

0 

0.013 

O.OCO 

0.019 

O.OI  1 

5 

0.032 

0.000 

0.0(X) 

0.01 1 

7 

0.013 

0.000 

0.000 

0,004 

Mean 

0.019 

0.000 

0.006 

0.009 
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